Nov 03 19:30:45.384342 ip-10-0-131-234 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:30:45.831299 ip-10-0-131-234 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:30:45.831299 ip-10-0-131-234 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:30:45.831299 ip-10-0-131-234 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:30:45.831299 ip-10-0-131-234 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:30:45.839233 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839129    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:45.839233 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839227    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:45.839233 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839231    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:45.839233 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839235    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:45.839233 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839238    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:45.839233 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839241    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:45.839233 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839245    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839248    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839251    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839255    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839257    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839260    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839263    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839266    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839269    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839271    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839274    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839277    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839279    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839281    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839285    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839288    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839290    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839293    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839296    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839298    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:45.839482 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839301    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839303    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839306    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839309    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839311    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839314    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839318    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839320    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839323    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839326    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839328    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839331    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839335    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839340    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839343    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839346    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839349    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839352    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839354    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:45.839946 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839358    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839360    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839364    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839367    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839370    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839372    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839375    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839378    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839380    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839383    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839386    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839390    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839395    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839398    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839401    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839404    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839406    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839409    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839412    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:45.840421 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839415    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839418    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839421    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839424    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839427    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839429    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839432    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839435    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839438    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839441    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839443    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839446    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839448    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839451    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839453    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839456    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839459    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839462    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839464    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839468    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:45.840874 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839471    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839474    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839476    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839479    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839482    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839486    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839489    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839492    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839495    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839498    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839501    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839504    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839507    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839510    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839513    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839515    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839518    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839521    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839524    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839527    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:45.841359 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839529    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.839532    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840412    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840421    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840424    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840427    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840431    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840434    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840437    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840439    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840442    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840445    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840448    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840451    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840454    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840457    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840459    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840464    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840468    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:45.841843 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840471    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840474    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840477    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840480    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840483    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840486    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840489    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840491    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840495    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840499    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840502    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840505    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840507    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840510    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840513    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840515    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840518    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840520    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840523    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:45.842354 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840526    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840529    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840531    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840534    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840536    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840539    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840541    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840544    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840547    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840550    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840552    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840555    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840557    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840560    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840563    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840566    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840569    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840571    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840574    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840577    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:45.842803 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840580    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840583    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840585    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840588    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840590    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840593    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840596    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840598    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840601    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840603    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840606    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840608    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840611    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840613    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840616    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840618    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840621    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840623    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840626    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840628    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:45.843293 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840631    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840634    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840636    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840640    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840642    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840645    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840647    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840650    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840653    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840655    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840658    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840661    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840664    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840666    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840669    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840671    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840674    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840677    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840679    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840683    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:45.843763 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840685    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840688    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840690    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840693    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840695    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840698    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840701    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840704    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840706    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.840709    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841248    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841254    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841259    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841262    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841265    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841268    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841271    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841274    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841277    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841279    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:45.844251 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841282    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841285    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841288    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841290    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841295    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841298    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841302    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841305    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841308    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841311    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841314    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841316    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841319    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841322    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841324    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841327    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841329    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841332    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841335    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:45.844734 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841338    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841340    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841343    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841345    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841348    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841351    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841353    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841356    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841359    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841361    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841364    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841367    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841370    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841372    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841375    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841378    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841380    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841383    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841386    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:45.845190 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841388    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841391    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841393    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841395    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841398    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841400    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841403    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841406    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841408    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841410    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841413    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841415    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841418    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841421    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841423    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841426    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841429    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841431    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841434    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841436    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:45.845639 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841440    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841443    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841446    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841448    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841450    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841453    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841456    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841459    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841461    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841464    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841466    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841469    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841471    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841474    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841476    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841479    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841481    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841483    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841487    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841498    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:45.846125 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841501    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841504    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841506    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841510    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841513    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841515    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841518    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841520    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841523    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841526    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841531    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841534    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841540    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841544    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841547    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841549    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841552    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.841554    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842688    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842698    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:30:45.846715 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842705    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842710    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842715    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842719    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842723    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842727    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842731    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842734    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842737    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842741    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842744    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842747    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842750    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842753    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842756    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842759    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842765    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842767    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842771    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842774    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842778    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842782    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842786    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:30:45.847207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842789    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842795    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842799    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842804    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842807    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842811    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842814    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842819    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842822    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842825    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842828    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842832    2576 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842835    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842839    2576 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842843    2576 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842845    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842849    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842852    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842856    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842859    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842863    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842866    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842870    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842873    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842877    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:30:45.847734 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842880    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842883    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842886    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842889    2576 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842893    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842896    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842899    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842903    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842906    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842909    2576 flags.go:64] FLAG: --help="false"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842912    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842916    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842920    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842923    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842926    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842929    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842932    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842935    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842938    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842941    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842944    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842947    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842950    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842953    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:30:45.848369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842955    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842958    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842961    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842964    2576 flags.go:64] FLAG: --lock-file=""
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842967    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842970    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842973    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842979    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842982    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842985    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.842998    2576 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843001    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843004    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843007    2576 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843010    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843015    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843019    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843022    2576 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843026    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843029    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843033    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843036    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843039    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843042    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843045    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:30:45.848942 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843052    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843055    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843058    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843061    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843064    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843067    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843070    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843073    2576 flags.go:64] FLAG: --port="10250"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843076    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843079    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0043e197a5031c2a6"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843082    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843085    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843088    2576 flags.go:64] FLAG: --register-node="true"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843091    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843095    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843098    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843101    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843104    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843109    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843112    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843115    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843118    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843121    2576 flags.go:64] FLAG: --runonce="false"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843124    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843127    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843131    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:30:45.849573 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843133    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843137    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843141    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843144    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843147    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843150    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843153    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843156    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843159    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843162    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843165    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843168    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843173    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843176    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843179    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843184    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843187    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843190    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843193    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843196    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843199    2576 flags.go:64] FLAG: --v="2"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843203    2576 flags.go:64] FLAG: --version="false"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843208    2576 flags.go:64] FLAG: --vmodule=""
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843212    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.843216    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:30:45.850335 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844135    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844141    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844145    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844148    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844151    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844153    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844156    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844159    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844161    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844164    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844167    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844170    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844173    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844176    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844178    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844181    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844184    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844186    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844189    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:45.850925 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844191    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844194    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844197    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844199    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844202    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844207    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844209    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844215    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844218    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844221    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844224    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844227    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844230    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844233    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844235    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844238    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844240    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844243    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844246    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844248    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:45.851400 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844251    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844254    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844257    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844260    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844263    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844266    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844268    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844271    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844273    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844276    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844279    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844281    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844284    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844286    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844289    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844292    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844294    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844298    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844301    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:45.851866 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844308    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844311    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844314    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844317    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844320    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844323    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844327    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844330    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844333    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844336    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844338    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844341    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844344    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844346    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844349    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844352    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844354    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844357    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844360    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:45.852338 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844364    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844369    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844372    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844375    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844378    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844380    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844383    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844386    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844388    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844391    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844394    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844398    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844401    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844405    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844407    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844410    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844413    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844416    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844418    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844421    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:45.852786 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844423    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844426    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844429    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844431    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844434    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844436    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844439    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844442    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844444    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.844474    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844512    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844516    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844519    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844522    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844525    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844528    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:45.853284 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844531    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844534    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844538    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844542    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844545    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844548    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844551    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844553    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844557    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844562    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844564    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844567    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844570    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844572    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844575    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844578    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844580    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844583    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844586    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:45.853674 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844589    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844591    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844594    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844596    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844599    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844602    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844605    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844608    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844610    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844613    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844617    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844619    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844622    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844624    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844627    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844630    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844632    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844635    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844637    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:45.854174 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844640    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844642    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844646    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844650    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844653    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844657    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844659    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844662    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844665    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844667    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844670    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844672    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844675    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844677    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844680    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844682    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844685    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844687    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844690    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:45.854625 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844693    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844695    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844698    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844700    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844703    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844707    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844709    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844712    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844714    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844717    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844719    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844722    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844724    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844727    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844730    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844733    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844738    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844741    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844744    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844746    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:45.855089 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844749    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844752    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844754    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844757    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844760    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844762    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844765    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844767    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844770    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844773    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844776    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844779    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844781    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844784    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844786    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844789    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844792    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844794    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844800    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:45.855556 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844803    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844806    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844809    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.844812    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.852297    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.852315    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.853099    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.853106    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853165    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853171    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853174    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853177    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853179    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853182    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853185    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853189    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853191    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853194    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853196    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:45.856056 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853199    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853202    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853205    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853207    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853212    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853216    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853220    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853223    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853226    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853228    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853231    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853234    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853236    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853239    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853241    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853244    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853247    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853249    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853254    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:45.856501 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853257    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853260    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853263    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853266    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853268    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853271    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853273    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853276    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853278    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853281    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853283    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853286    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853289    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853292    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853295    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853297    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853300    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853303    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853306    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:45.856947 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853308    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853311    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853313    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853316    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853319    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853321    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853324    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853326    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853329    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853332    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853334    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853337    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853340    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853344    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853346    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853349    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853352    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853354    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853357    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853359    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:45.857415 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853362    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853365    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853367    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853370    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853372    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853375    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853377    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853380    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853382    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853385    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853388    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853390    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853393    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853395    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853398    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853400    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853403    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853406    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853408    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:45.857896 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853412    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853416    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853419    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853421    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853423    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853427    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853430    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853433    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853436    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853439    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853441    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853444    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853447    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853450    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853452    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853455    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853458    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:45.858424 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853460    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.853498    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853643    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853647    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853650    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853653    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853656    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853658    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853661    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853663    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853666    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853669    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853672    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853674    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853677    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853679    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:45.858829 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853682    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853685    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853688    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853690    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853694    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853697    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853699    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853702    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853705    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853707    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853710    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853712    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853715    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853717    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853720    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853722    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853725    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853727    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853730    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853732    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:45.859218 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853735    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853737    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853740    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853742    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853745    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853748    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853750    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853753    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853755    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853758    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853761    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853763    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853765    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853768    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853770    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853773    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853776    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853779    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853781    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853784    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:45.859690 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853787    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853791    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853795    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853797    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853800    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853803    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853806    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853808    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853811    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853813    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853816    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853818    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853821    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853823    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853826    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853828    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853831    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853833    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:45.860170 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853837    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853841    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853844    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853847    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853850    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853852    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853855    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853857    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853860    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853862    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853865    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853868    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853870    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853873    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853876    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853879    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853882    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853884    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853887    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853889    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:45.860636 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853892    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853894    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853897    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853899    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853902    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853905    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853907    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853910    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853912    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853914    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853917    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853920    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853922    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853925    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.853947    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:30:45.861122 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853979    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853982    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.853985    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854002    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854005    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854008    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854011    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854014    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854016    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854019    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854022    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854025    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854027    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854031    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854033    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854036    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854039    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854041    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854044    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854047    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:45.861498 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854050    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854052    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854055    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854058    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854061    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854064    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854067    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854070    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854072    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854075    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854078    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854081    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854083    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854086    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854089    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854091    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854094    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854097    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854099    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854103    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:45.862186 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854107    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854110    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854112    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854115    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854118    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854121    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854124    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854127    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854129    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854132    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854134    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854137    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854140    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854142    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854145    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854147    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854151    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854155    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:45.862692 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854159    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854161    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854164    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854166    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854169    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854172    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854175    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854177    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854180    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854183    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854186    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854189    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854192    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854194    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854197    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854200    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854202    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854205    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854207    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854210    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:45.863149 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854213    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854216    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854219    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854221    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854224    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854227    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854229    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854232    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854234    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854237    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854239    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854245    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854247    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854250    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854253    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854255    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854258    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854261    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854263    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:45.863612 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854266    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854269    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854272    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854275    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854277    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854280    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854282    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854285    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:45.854287    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.854342    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.856917    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.858046    2576 server.go:1014] "Starting client certificate rotation"
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.858144    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.858181    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.860276    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:30:45.864166 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.861546    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:30:45.875703 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.875433    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:30:45.880643 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.880627    2576 log.go:25] "Validated CRI v1 image API"
Nov 03 19:30:45.881437 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.881410    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:30:45.884330 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.884302    2576 fs.go:135] Filesystem UUIDs: map[1e235f90-949f-48e7-aca8-d8d017ee165b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e03333df-c179-4a4d-8e46-8ed103713010:/dev/nvme0n1p3]
Nov 03 19:30:45.884403 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.884327    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 19:30:45.887983 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.887966    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:30:45.890305 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.890176    2576 manager.go:217] Machine: {Timestamp:2026-11-03 19:30:45.888647559 +0000 UTC m=+0.378381454 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099386 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a09d085c1a356e076734ee67ac289 SystemUUID:ec2a09d0-85c1-a356-e076-734ee67ac289 BootID:865fa0b7-6663-4b22-b9fe-52cc48348515 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f4:30:cf:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f4:30:cf:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:74:84:ae:89:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:30:45.890305 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.890307    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:30:45.890459 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.890443    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:30:45.894133 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.894105    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:30:45.894293 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.894136    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-234.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:30:45.894339 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.894295    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:30:45.894339 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.894303    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:30:45.894393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.894356    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:30:45.894393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.894376    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:30:45.895716 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.895705    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:30:45.899665 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.899652    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:30:45.899782 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.899773    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:30:45.899810 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.899804    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:30:45.899842 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.899815    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:30:45.899842 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.899825    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:30:45.899842 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.899835    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:30:45.899919 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.899843    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:30:45.900248 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.900234    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:30:45.902227 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.902211    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ptl9f"
Nov 03 19:30:45.902771 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.902761    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:30:45.909614 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.909588    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ptl9f"
Nov 03 19:30:45.909700 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.909688    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:30:45.910661 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.910648    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:30:45.910698 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.910692    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:30:45.910881 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.910871    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:30:45.910914 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.910886    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:30:45.910914 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.910892    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:30:45.910914 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.910897    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:30:45.910914 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.910903    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:30:45.910914 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.910909    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:30:45.911662 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.911651    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:30:45.911695 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.911663    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:30:45.911695 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.911671    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:30:45.911695 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.911678    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:30:45.911695 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.911692    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:30:45.911814 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.911701    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:30:45.913249 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.913239    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:30:45.913285 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.913250    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:30:45.917653 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.917639    2576 server.go:1263] "Started kubelet"
Nov 03 19:30:45.917755 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.917715    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:30:45.917807 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.917724    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:30:45.917863 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.917815    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:30:45.918539 ip-10-0-131-234 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:30:45.918906 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.918890    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:30:45.922764 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.922674    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:30:45.925778 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:45.925754    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 19:30:45.929282 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.929263    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:45.930902 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.930885    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:30:45.931724 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.931705    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:30:45.931862 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:45.931842    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-234.us-east-2.compute.internal\" not found"
Nov 03 19:30:45.932253 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932214    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-234.us-east-2.compute.internal" not found
Nov 03 19:30:45.932415 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932393    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:45.932670 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932651    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:30:45.932670 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932668    2576 factory.go:153] Registering CRI-O factory
Nov 03 19:30:45.932765 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932683    2576 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:30:45.932765 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932651    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:30:45.932765 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932753    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:30:45.932852 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932738    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:30:45.932852 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932801    2576 factory.go:55] Registering systemd factory
Nov 03 19:30:45.932852 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932810    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:30:45.932852 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932830    2576 factory.go:103] Registering Raw factory
Nov 03 19:30:45.932852 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932842    2576 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:30:45.932852 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932843    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:30:45.932852 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.932853    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:30:45.934390 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.934368    2576 manager.go:319] Starting recovery of all containers
Nov 03 19:30:45.935807 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.935790    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:45.938786 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:45.938569    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-234.us-east-2.compute.internal\" not found" node="ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:45.944207 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.944190    2576 manager.go:324] Recovery completed
Nov 03 19:30:45.946416 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.946393    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-234.us-east-2.compute.internal" not found
Nov 03 19:30:45.948831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.948818    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:45.950558 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.950540    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:45.950653 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.950577    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:45.950653 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.950593    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:45.951223 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.951209    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:30:45.951223 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.951218    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:45.951335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.951220    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:30:45.951335 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.951260    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:30:45.952073 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.952055    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:45.952073 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.952080    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:45.952199 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.952091    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:45.952575 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.952563    2576 policy_none.go:50] "Start"
Nov 03 19:30:45.952642 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.952589    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:30:45.952642 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.952603    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:30:45.953737 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.953723    2576 policy_none.go:44] "Start"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.990594    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.994798    2576 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:45.994833    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.994845    2576 server.go:87] "Starting device plugin registration server"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.995169    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.995181    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.995288    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.995382    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:45.995391    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:45.995776    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:45.995833    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-234.us-east-2.compute.internal\" not found"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.000399    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.001507    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.001531    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.001549    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:46.001578    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.005161    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-234.us-east-2.compute.internal" not found
Nov 03 19:30:46.006866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.005615    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:46.096175 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.096046    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:46.097122 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.097104    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:46.097209 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.097160    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:46.097209 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.097172    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:46.097209 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.097195    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.102346 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.102327    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"]
Nov 03 19:30:46.102444 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.102381    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:46.103254 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.103239    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:46.103332 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.103265    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:46.103332 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.103276    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:46.105487 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.105475    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:46.105642 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.105627    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.105679 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.105658    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:46.106173 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.106156    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:46.106173 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.106163    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:46.106288 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.106187    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:46.106288 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.106188    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:46.106288 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.106232    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:46.106288 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.106209    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:46.107666 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.107648    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.108471 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.108458    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.132732 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.132710    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.134966 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.134946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9cc75c72d3d8481cfe8602c9dd164cf-config\") pod \"kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"d9cc75c72d3d8481cfe8602c9dd164cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.135078 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.134972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.135078 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.135012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.145791 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.145775    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.146061 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.146044    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:30:46.146175 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.146162    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.148570 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.148557    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.155438 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.155423    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:30:46.162097 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.162084    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:30:46.162097 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.162092    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:30:46.162209 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:46.162118    2576 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.162209 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:46.162124    2576 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.235512 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.235480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9cc75c72d3d8481cfe8602c9dd164cf-config\") pod \"kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"d9cc75c72d3d8481cfe8602c9dd164cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.235695 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.235514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.235695 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.235573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.235695 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.235598    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9cc75c72d3d8481cfe8602c9dd164cf-config\") pod \"kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"d9cc75c72d3d8481cfe8602c9dd164cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.235695 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.235600    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.235695 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.235607    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.468432 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.468338    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:46.468432 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:46.468362    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:30:48.218918 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.218729    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 19:30:48.219531 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.219051    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:30:48.219531 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.219055    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:30:48.219531 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.219055    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:30:48.262147 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.262117    2576 apiserver.go:52] "Watching apiserver"
Nov 03 19:30:48.266875 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.266848    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:30:48.267162 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.267138    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-dqz7h","kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal","openshift-multus/multus-vbrfn","openshift-dns/node-resolver-q22m7","openshift-network-diagnostics/network-check-target-m4rw5","openshift-network-operator/iptables-alerter-qmfcq","kube-system/global-pull-secret-syncer-vj6pd","kube-system/konnectivity-agent-4jl9f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2","openshift-multus/multus-additional-cni-plugins-w2kqz","openshift-multus/network-metrics-daemon-mgl8h","openshift-cluster-node-tuning-operator/tuned-mfjxm","openshift-image-registry/node-ca-gcvlj"]
Nov 03 19:30:48.269832 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.269815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.271859 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.271836    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.273050 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.273024    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:30:48.273154 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.273061    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:30:48.273154 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.273126    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:30:48.273278 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.273208    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:30:48.273278 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.273258    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:30:48.273278 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.273262    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-glk7q\""
Nov 03 19:30:48.273398 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.273321    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:30:48.275193 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.274261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:30:48.275193 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.274529    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:30:48.275193 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.274793    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vgldx\""
Nov 03 19:30:48.275193 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.274941    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:30:48.275193 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.274984    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:30:48.275452 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.275233    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:30:48.276371 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.276304    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:25:47 +0000 UTC" deadline="2028-06-01 05:03:03.495907834 +0000 UTC"
Nov 03 19:30:48.276371 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.276371    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13809h32m15.219541016s"
Nov 03 19:30:48.276611 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.276592    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:30:48.276884 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.276867    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:30:48.276961 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.276868    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:48.277373 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.277343    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:30:48.277471 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.277385    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k6x7r\""
Nov 03 19:30:48.279598 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.279578    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:30:48.281687 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.281673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:48.281777 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.281719    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:30:48.282928 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.282910    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:30:48.283903 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.283885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:30:48.286167 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.286148    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.288501 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.288485    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.290636 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.290619    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:48.290726 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.290685    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:30:48.292502 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.292485    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 19:30:48.292866 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.292849    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.295054 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.295033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:30:48.299964 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.299947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-node-log\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.300104 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.299971    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-daemon-config\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.300104 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.299989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-multus-certs\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.300104 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-konnectivity-ca\") pod \"konnectivity-agent-4jl9f\" (UID: \"0a4d21a2-f237-4ffb-88c9-1855e8b4b887\") " pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:30:48.300104 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-bin\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.300104 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300037    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:30:48.300104 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-netd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.300403 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-var-lib-kubelet\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.300403 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-socket-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.300403 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.300403 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300271    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-kubelet\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.300403 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-env-overrides\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.300403 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-script-lib\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.300403 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-etc-kubernetes\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.300403 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-kubernetes\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-netns\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cni-binary-copy\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-run\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300608    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cnibin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-bin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34118633-5be6-49c0-9663-8c66046d847a-host-slash\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-systemd-units\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-conf\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.300831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-hostroot\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bzk\" (UniqueName: \"kubernetes.io/projected/448841dc-bb08-4989-8180-abe966bcaab4-kube-api-access-k4bzk\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-systemd\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300885    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34118633-5be6-49c0-9663-8c66046d847a-iptables-alerter-script\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-cnibin\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mfrm\" (UniqueName: \"kubernetes.io/projected/ad438ca5-2f83-4e9b-8058-69edd81f888e-kube-api-access-6mfrm\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.300999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-netns\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6xff\" (UniqueName: \"kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff\") pod \"network-check-target-m4rw5\" (UID: \"8f387957-9771-467e-9053-994d10fc652a\") " pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-modprobe-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-registration-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-system-cni-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-host\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-log-socket\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.301603 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-sys\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-device-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301411    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-slash\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-config\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ppqh\" (UniqueName: \"kubernetes.io/projected/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-kube-api-access-6ppqh\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysconfig\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvc7b\" (UniqueName: \"kubernetes.io/projected/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-kube-api-access-wvc7b\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-system-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-socket-dir-parent\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301654    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-k8s-cni-cncf-io\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-kubelet\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttftp\" (UniqueName: \"kubernetes.io/projected/8c63d965-5a4f-481a-8b25-0ac731629615-kube-api-access-ttftp\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-agent-certs\") pod \"konnectivity-agent-4jl9f\" (UID: \"0a4d21a2-f237-4ffb-88c9-1855e8b4b887\") " pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-etc-tuned\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r9kw\" (UniqueName: \"kubernetes.io/projected/34118633-5be6-49c0-9663-8c66046d847a-kube-api-access-8r9kw\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:30:48.302473 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-etc-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-ovn\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-conf-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7srz\" (UniqueName: \"kubernetes.io/projected/1697fe5f-c2b8-46e3-b57d-73afd75e6926-kube-api-access-z7srz\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c63d965-5a4f-481a-8b25-0ac731629615-tmp-dir\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.301967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-kubelet-config\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-dbus\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-var-lib-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovn-node-metrics-cert\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302110    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c63d965-5a4f-481a-8b25-0ac731629615-hosts-file\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-lib-modules\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-host\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302211    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-tmp\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-multus\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtjc\" (UniqueName: \"kubernetes.io/projected/c455037a-fc47-4c3d-a917-854f924a5df0-kube-api-access-frtjc\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.303393 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-sys-fs\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.303823 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbldx\" (UniqueName: \"kubernetes.io/projected/44c7af65-aff0-41b1-8f98-2adf11d53954-kube-api-access-nbldx\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.303823 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302374    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-serviceca\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:30:48.303823 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.302410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-systemd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.317144 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.317120    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pr7tz"
Nov 03 19:30:48.324154 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.324128    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pr7tz"
Nov 03 19:30:48.376352 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376332    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:30:48.376890 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376861    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:30:48.376890 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376876    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pq7nf\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376884    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376877    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6p8wh\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376912    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wllhj\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376912    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376984    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kqxn2\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376994    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376999    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376958    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.377021    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376979    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376997    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376947    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376941    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376952    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:30:48.377059 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.376968    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:30:48.377522 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.377023    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bh6bb\""
Nov 03 19:30:48.377522 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.377081    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9h8df\""
Nov 03 19:30:48.395305 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.395281    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:30:48.403120 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.403256 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-log-socket\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.403256 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.403256 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-sys\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.403256 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-device-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.403256 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403206    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.403256 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403223    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-slash\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-config\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403279    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ppqh\" (UniqueName: \"kubernetes.io/projected/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-kube-api-access-6ppqh\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-sys\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403316    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-device-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysconfig\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403332    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-log-socket\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403357    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-slash\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403370    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvc7b\" (UniqueName: \"kubernetes.io/projected/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-kube-api-access-wvc7b\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403395    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysconfig\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-system-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-socket-dir-parent\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-k8s-cni-cncf-io\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403445    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-system-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403449    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-kubelet\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403469    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-kubelet\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.403557 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttftp\" (UniqueName: \"kubernetes.io/projected/8c63d965-5a4f-481a-8b25-0ac731629615-kube-api-access-ttftp\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-agent-certs\") pod \"konnectivity-agent-4jl9f\" (UID: \"0a4d21a2-f237-4ffb-88c9-1855e8b4b887\") " pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403533    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-etc-tuned\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r9kw\" (UniqueName: \"kubernetes.io/projected/34118633-5be6-49c0-9663-8c66046d847a-kube-api-access-8r9kw\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-etc-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-ovn\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-conf-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403655    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7srz\" (UniqueName: \"kubernetes.io/projected/1697fe5f-c2b8-46e3-b57d-73afd75e6926-kube-api-access-z7srz\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c63d965-5a4f-481a-8b25-0ac731629615-tmp-dir\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403698    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-kubelet-config\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-dbus\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403766    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-config\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-var-lib-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovn-node-metrics-cert\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c63d965-5a4f-481a-8b25-0ac731629615-hosts-file\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:30:48.404405 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-lib-modules\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-host\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403925    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-etc-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-tmp\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-multus\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtjc\" (UniqueName: \"kubernetes.io/projected/c455037a-fc47-4c3d-a917-854f924a5df0-kube-api-access-frtjc\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-sys-fs\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbldx\" (UniqueName: \"kubernetes.io/projected/44c7af65-aff0-41b1-8f98-2adf11d53954-kube-api-access-nbldx\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-serviceca\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-systemd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404089    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-node-log\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-daemon-config\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404119    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-multus-certs\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-konnectivity-ca\") pod \"konnectivity-agent-4jl9f\" (UID: \"0a4d21a2-f237-4ffb-88c9-1855e8b4b887\") " pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-bin\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-netd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-var-lib-kubelet\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.405136 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404212    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-ovn\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404243    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-socket-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404248    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-conf-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404277    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-multus\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-kubelet\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404291    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-env-overrides\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404447    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-sys-fs\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-script-lib\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-etc-kubernetes\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-kubernetes\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-netns\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404694    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-multus-certs\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.405752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.403510    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-k8s-cni-cncf-io\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.404638    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-env-overrides\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405286    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-konnectivity-ca\") pod \"konnectivity-agent-4jl9f\" (UID: \"0a4d21a2-f237-4ffb-88c9-1855e8b4b887\") " pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cni-binary-copy\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-run\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cnibin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-bin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34118633-5be6-49c0-9663-8c66046d847a-host-slash\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-systemd-units\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-conf\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-hostroot\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bzk\" (UniqueName: \"kubernetes.io/projected/448841dc-bb08-4989-8180-abe966bcaab4-kube-api-access-k4bzk\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-systemd\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34118633-5be6-49c0-9663-8c66046d847a-iptables-alerter-script\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:30:48.406510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-cnibin\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mfrm\" (UniqueName: \"kubernetes.io/projected/ad438ca5-2f83-4e9b-8058-69edd81f888e-kube-api-access-6mfrm\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-netns\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405733    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xff\" (UniqueName: \"kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff\") pod \"network-check-target-m4rw5\" (UID: \"8f387957-9771-467e-9053-994d10fc652a\") " pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-modprobe-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-registration-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-system-cni-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405869    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-host\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405943    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-hostroot\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405961    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-host\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.405968    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-systemd\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406056    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-modprobe-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406134    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-registration-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406312    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406319    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-system-cni-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406390    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-cnibin\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.406421    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:48.407399 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406436    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.406500    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs podName:448841dc-bb08-4989-8180-abe966bcaab4 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:48.906477019 +0000 UTC m=+2.035028897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs") pod "network-metrics-daemon-mgl8h" (UID: "448841dc-bb08-4989-8180-abe966bcaab4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406540    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-conf\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406663    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34118633-5be6-49c0-9663-8c66046d847a-iptables-alerter-script\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406819    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-socket-dir-parent\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406878    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-kubelet-config\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406923    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-systemd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.406965    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-node-log\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407056    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-socket-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407110    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-dbus\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407132    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407217    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-bin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407278    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-run\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cni-binary-copy\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407333    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cnibin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407564    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-daemon-config\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407566    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-netns\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.408365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407606    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-bin\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407635    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-var-lib-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407644    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-netns\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407686    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-kubelet\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407697    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-var-lib-kubelet\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407708    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c63d965-5a4f-481a-8b25-0ac731629615-tmp-dir\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407743    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-kubernetes\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407762    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407763    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-systemd-units\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407807    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34118633-5be6-49c0-9663-8c66046d847a-host-slash\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407810    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c63d965-5a4f-481a-8b25-0ac731629615-hosts-file\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407832    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-host\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407903    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.407968    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-lib-modules\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.408230    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-netd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.408454    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-serviceca\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.408480    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.408839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.409233 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.408935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-script-lib\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.410081 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.409008    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:48.410081 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.409012    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-tmp\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.410081 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.409060    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret podName:319a92b5-37b5-40b2-88cd-5550aab72650 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:48.909043304 +0000 UTC m=+2.037595156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret") pod "global-pull-secret-syncer-vj6pd" (UID: "319a92b5-37b5-40b2-88cd-5550aab72650") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:48.410081 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.409427    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-agent-certs\") pod \"konnectivity-agent-4jl9f\" (UID: \"0a4d21a2-f237-4ffb-88c9-1855e8b4b887\") " pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:30:48.410081 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.409560    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.412419 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.412393    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-etc-tuned\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.413019 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.412995    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:30:48.413112 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.413024    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:30:48.413112 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.413038    2576 projected.go:196] Error preparing data for projected volume kube-api-access-w6xff for pod openshift-network-diagnostics/network-check-target-m4rw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:48.413112 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.413094    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff podName:8f387957-9771-467e-9053-994d10fc652a nodeName:}" failed. No retries permitted until 2026-11-03 19:30:48.913076072 +0000 UTC m=+2.041627947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w6xff" (UniqueName: "kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff") pod "network-check-target-m4rw5" (UID: "8f387957-9771-467e-9053-994d10fc652a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:48.413901 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.413873    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r9kw\" (UniqueName: \"kubernetes.io/projected/34118633-5be6-49c0-9663-8c66046d847a-kube-api-access-8r9kw\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:30:48.414058 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.414034    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtjc\" (UniqueName: \"kubernetes.io/projected/c455037a-fc47-4c3d-a917-854f924a5df0-kube-api-access-frtjc\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.414253 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.414233    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttftp\" (UniqueName: \"kubernetes.io/projected/8c63d965-5a4f-481a-8b25-0ac731629615-kube-api-access-ttftp\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:30:48.415115 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.415093    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvc7b\" (UniqueName: \"kubernetes.io/projected/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-kube-api-access-wvc7b\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:30:48.415548 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.415516    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ppqh\" (UniqueName: \"kubernetes.io/projected/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-kube-api-access-6ppqh\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.415648 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.415623    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7srz\" (UniqueName: \"kubernetes.io/projected/1697fe5f-c2b8-46e3-b57d-73afd75e6926-kube-api-access-z7srz\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.416218 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.416150    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovn-node-metrics-cert\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.416801 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.416743    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbldx\" (UniqueName: \"kubernetes.io/projected/44c7af65-aff0-41b1-8f98-2adf11d53954-kube-api-access-nbldx\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.417250 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.417231    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bzk\" (UniqueName: \"kubernetes.io/projected/448841dc-bb08-4989-8180-abe966bcaab4-kube-api-access-k4bzk\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:48.418344 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.418324    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mfrm\" (UniqueName: \"kubernetes.io/projected/ad438ca5-2f83-4e9b-8058-69edd81f888e-kube-api-access-6mfrm\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.419275 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.419253    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-etc-kubernetes\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.429633 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.429611    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:30:48.432485 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.432472    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:30:48.443282 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.443261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:30:48.535371 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:48.535336    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cc75c72d3d8481cfe8602c9dd164cf.slice/crio-4c63ffbd45c6598a06ba4c7a38bd2f6500230e2a127db1cdad004a6cd8b551e2 WatchSource:0}: Error finding container 4c63ffbd45c6598a06ba4c7a38bd2f6500230e2a127db1cdad004a6cd8b551e2: Status 404 returned error can't find the container with id 4c63ffbd45c6598a06ba4c7a38bd2f6500230e2a127db1cdad004a6cd8b551e2
Nov 03 19:30:48.535842 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:48.535818    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b103dd723deeb9c86a7ed93f39fb9c.slice/crio-e4e8c086238cfe606828d2c4c2b7eed7db3f8ba5d6e960919ea42e279814161d WatchSource:0}: Error finding container e4e8c086238cfe606828d2c4c2b7eed7db3f8ba5d6e960919ea42e279814161d: Status 404 returned error can't find the container with id e4e8c086238cfe606828d2c4c2b7eed7db3f8ba5d6e960919ea42e279814161d
Nov 03 19:30:48.541672 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.541651    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:30:48.618135 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.618109    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:30:48.625243 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.625219    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vbrfn"
Nov 03 19:30:48.628650 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:48.628627    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdd5ccb_e17b_4bd6_adb3_30c82c2836c1.slice/crio-7dfe88d7dd01e5ef2eed0e162a001efce1646b7dd90739005770c285a18a6a6f WatchSource:0}: Error finding container 7dfe88d7dd01e5ef2eed0e162a001efce1646b7dd90739005770c285a18a6a6f: Status 404 returned error can't find the container with id 7dfe88d7dd01e5ef2eed0e162a001efce1646b7dd90739005770c285a18a6a6f
Nov 03 19:30:48.634275 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.634252    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:30:48.634563 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:48.634540    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1697fe5f_c2b8_46e3_b57d_73afd75e6926.slice/crio-d3e76e61fa30d5b74144b99cbe768ed6e805d24628f2f34a87e4d2b065e6881e WatchSource:0}: Error finding container d3e76e61fa30d5b74144b99cbe768ed6e805d24628f2f34a87e4d2b065e6881e: Status 404 returned error can't find the container with id d3e76e61fa30d5b74144b99cbe768ed6e805d24628f2f34a87e4d2b065e6881e
Nov 03 19:30:48.643528 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:48.643506    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c63d965_5a4f_481a_8b25_0ac731629615.slice/crio-108ce33e60df30154d94da41931e07b9c95caee552fd94e5dff30a69cc8a902b WatchSource:0}: Error finding container 108ce33e60df30154d94da41931e07b9c95caee552fd94e5dff30a69cc8a902b: Status 404 returned error can't find the container with id 108ce33e60df30154d94da41931e07b9c95caee552fd94e5dff30a69cc8a902b
Nov 03 19:30:48.653114 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.653095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:30:48.662739 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:48.662710    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34118633_5be6_49c0_9663_8c66046d847a.slice/crio-13ea335b9679fd32d02a55062b8596cff1372427eb952cf77fb6b4831b5a8dbe WatchSource:0}: Error finding container 13ea335b9679fd32d02a55062b8596cff1372427eb952cf77fb6b4831b5a8dbe: Status 404 returned error can't find the container with id 13ea335b9679fd32d02a55062b8596cff1372427eb952cf77fb6b4831b5a8dbe
Nov 03 19:30:48.670890 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.670869    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:30:48.680397 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:48.680368    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4d21a2_f237_4ffb_88c9_1855e8b4b887.slice/crio-eca8f0b225ae055628e6816ad6c0cfd08ac606a88a32ac697ffbb024772605ca WatchSource:0}: Error finding container eca8f0b225ae055628e6816ad6c0cfd08ac606a88a32ac697ffbb024772605ca: Status 404 returned error can't find the container with id eca8f0b225ae055628e6816ad6c0cfd08ac606a88a32ac697ffbb024772605ca
Nov 03 19:30:48.691397 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.691376    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:30:48.700534 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:48.700507    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c7af65_aff0_41b1_8f98_2adf11d53954.slice/crio-382413560c9799898e204792bd056dc10a8edc5094dd006f2a3b9a3e78e9ab7d WatchSource:0}: Error finding container 382413560c9799898e204792bd056dc10a8edc5094dd006f2a3b9a3e78e9ab7d: Status 404 returned error can't find the container with id 382413560c9799898e204792bd056dc10a8edc5094dd006f2a3b9a3e78e9ab7d
Nov 03 19:30:48.836272 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:48.836240    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc455037a_fc47_4c3d_a917_854f924a5df0.slice/crio-7037a0c9d8524c2d44eafeffb65654aa22af484c24b5f38d8472f51cc607a9df WatchSource:0}: Error finding container 7037a0c9d8524c2d44eafeffb65654aa22af484c24b5f38d8472f51cc607a9df: Status 404 returned error can't find the container with id 7037a0c9d8524c2d44eafeffb65654aa22af484c24b5f38d8472f51cc607a9df
Nov 03 19:30:48.882062 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:48.882033    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad438ca5_2f83_4e9b_8058_69edd81f888e.slice/crio-86baadc1f9343dc7e0c95a6855914e37e2fe2fed21b8393f93cbc95b50c11fed WatchSource:0}: Error finding container 86baadc1f9343dc7e0c95a6855914e37e2fe2fed21b8393f93cbc95b50c11fed: Status 404 returned error can't find the container with id 86baadc1f9343dc7e0c95a6855914e37e2fe2fed21b8393f93cbc95b50c11fed
Nov 03 19:30:48.901207 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:30:48.901163    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadabc7a3_99a6_43f0_b1ed_8a0395d310a4.slice/crio-19c150003ec54bbf8a4ebcdb038974342b4f635d92ee2cffb0fb4bcd864abfa2 WatchSource:0}: Error finding container 19c150003ec54bbf8a4ebcdb038974342b4f635d92ee2cffb0fb4bcd864abfa2: Status 404 returned error can't find the container with id 19c150003ec54bbf8a4ebcdb038974342b4f635d92ee2cffb0fb4bcd864abfa2
Nov 03 19:30:48.912956 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.912932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:48.913053 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:48.912975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:48.913123 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.913064    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:48.913123 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.913072    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:48.913123 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.913109    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs podName:448841dc-bb08-4989-8180-abe966bcaab4 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:49.913094919 +0000 UTC m=+3.041646772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs") pod "network-metrics-daemon-mgl8h" (UID: "448841dc-bb08-4989-8180-abe966bcaab4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:48.913256 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:48.913134    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret podName:319a92b5-37b5-40b2-88cd-5550aab72650 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:49.913114517 +0000 UTC m=+3.041666389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret") pod "global-pull-secret-syncer-vj6pd" (UID: "319a92b5-37b5-40b2-88cd-5550aab72650") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:49.013688 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.013651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xff\" (UniqueName: \"kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff\") pod \"network-check-target-m4rw5\" (UID: \"8f387957-9771-467e-9053-994d10fc652a\") " pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:49.013873 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:49.013814    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:30:49.013873 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:49.013844    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:30:49.013873 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:49.013856    2576 projected.go:196] Error preparing data for projected volume kube-api-access-w6xff for pod openshift-network-diagnostics/network-check-target-m4rw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:49.014029 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:49.013923    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff podName:8f387957-9771-467e-9053-994d10fc652a nodeName:}" failed. No retries permitted until 2026-11-03 19:30:50.013903247 +0000 UTC m=+3.142455114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6xff" (UniqueName: "kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff") pod "network-check-target-m4rw5" (UID: "8f387957-9771-467e-9053-994d10fc652a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:49.328654 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.328589    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:25:48 +0000 UTC" deadline="2028-07-09 21:13:25.943286428 +0000 UTC"
Nov 03 19:30:49.328654 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.328652    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14737h42m36.614638879s"
Nov 03 19:30:49.338532 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.338502    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:49.379788 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.379723    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q22m7" event={"ID":"8c63d965-5a4f-481a-8b25-0ac731629615","Type":"ContainerStarted","Data":"108ce33e60df30154d94da41931e07b9c95caee552fd94e5dff30a69cc8a902b"}
Nov 03 19:30:49.387707 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.387668    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gcvlj" event={"ID":"adabc7a3-99a6-43f0-b1ed-8a0395d310a4","Type":"ContainerStarted","Data":"19c150003ec54bbf8a4ebcdb038974342b4f635d92ee2cffb0fb4bcd864abfa2"}
Nov 03 19:30:49.396235 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.396135    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mfjxm" event={"ID":"c455037a-fc47-4c3d-a917-854f924a5df0","Type":"ContainerStarted","Data":"7037a0c9d8524c2d44eafeffb65654aa22af484c24b5f38d8472f51cc607a9df"}
Nov 03 19:30:49.400098 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.399906    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4jl9f" event={"ID":"0a4d21a2-f237-4ffb-88c9-1855e8b4b887","Type":"ContainerStarted","Data":"eca8f0b225ae055628e6816ad6c0cfd08ac606a88a32ac697ffbb024772605ca"}
Nov 03 19:30:49.405654 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.405581    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vbrfn" event={"ID":"1697fe5f-c2b8-46e3-b57d-73afd75e6926","Type":"ContainerStarted","Data":"d3e76e61fa30d5b74144b99cbe768ed6e805d24628f2f34a87e4d2b065e6881e"}
Nov 03 19:30:49.408787 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.408663    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h" event={"ID":"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1","Type":"ContainerStarted","Data":"7dfe88d7dd01e5ef2eed0e162a001efce1646b7dd90739005770c285a18a6a6f"}
Nov 03 19:30:49.413129 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.413053    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal" event={"ID":"d9cc75c72d3d8481cfe8602c9dd164cf","Type":"ContainerStarted","Data":"4c63ffbd45c6598a06ba4c7a38bd2f6500230e2a127db1cdad004a6cd8b551e2"}
Nov 03 19:30:49.415683 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.415653    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal" event={"ID":"50b103dd723deeb9c86a7ed93f39fb9c","Type":"ContainerStarted","Data":"e4e8c086238cfe606828d2c4c2b7eed7db3f8ba5d6e960919ea42e279814161d"}
Nov 03 19:30:49.420370 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.420326    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2kqz" event={"ID":"ad438ca5-2f83-4e9b-8058-69edd81f888e","Type":"ContainerStarted","Data":"86baadc1f9343dc7e0c95a6855914e37e2fe2fed21b8393f93cbc95b50c11fed"}
Nov 03 19:30:49.422922 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.422853    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2" event={"ID":"44c7af65-aff0-41b1-8f98-2adf11d53954","Type":"ContainerStarted","Data":"382413560c9799898e204792bd056dc10a8edc5094dd006f2a3b9a3e78e9ab7d"}
Nov 03 19:30:49.427979 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.427905    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qmfcq" event={"ID":"34118633-5be6-49c0-9663-8c66046d847a","Type":"ContainerStarted","Data":"13ea335b9679fd32d02a55062b8596cff1372427eb952cf77fb6b4831b5a8dbe"}
Nov 03 19:30:49.712453 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.712407    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:49.713479 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.713455    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:49.920662 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.920629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:49.920875 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:49.920687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:49.920875 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:49.920782    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:49.920875 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:49.920829    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret podName:319a92b5-37b5-40b2-88cd-5550aab72650 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:51.92081544 +0000 UTC m=+5.049367292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret") pod "global-pull-secret-syncer-vj6pd" (UID: "319a92b5-37b5-40b2-88cd-5550aab72650") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:49.921152 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:49.921129    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:49.921253 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:49.921200    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs podName:448841dc-bb08-4989-8180-abe966bcaab4 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:51.92118355 +0000 UTC m=+5.049735420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs") pod "network-metrics-daemon-mgl8h" (UID: "448841dc-bb08-4989-8180-abe966bcaab4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:50.022060 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:50.021368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xff\" (UniqueName: \"kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff\") pod \"network-check-target-m4rw5\" (UID: \"8f387957-9771-467e-9053-994d10fc652a\") " pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:50.022060 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:50.021576    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:30:50.022060 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:50.021595    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:30:50.022060 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:50.021608    2576 projected.go:196] Error preparing data for projected volume kube-api-access-w6xff for pod openshift-network-diagnostics/network-check-target-m4rw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:50.022060 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:50.021668    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff podName:8f387957-9771-467e-9053-994d10fc652a nodeName:}" failed. No retries permitted until 2026-11-03 19:30:52.021651116 +0000 UTC m=+5.150202978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6xff" (UniqueName: "kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff") pod "network-check-target-m4rw5" (UID: "8f387957-9771-467e-9053-994d10fc652a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:50.329021 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:50.328927    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:25:48 +0000 UTC" deadline="2028-05-02 19:51:51.378369985 +0000 UTC"
Nov 03 19:30:50.329021 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:50.328973    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13104h21m1.049402583s"
Nov 03 19:30:50.363478 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:50.363444    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:50.363696 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:50.363583    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:30:50.364016 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:50.363994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:50.364192 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:50.364096    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:30:50.364192 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:50.364159    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:50.364349 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:50.364255    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:30:51.941361 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:51.940216    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:51.941361 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:51.940299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:51.941361 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:51.940460    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:51.941361 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:51.940522    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs podName:448841dc-bb08-4989-8180-abe966bcaab4 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:55.940505144 +0000 UTC m=+9.069057014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs") pod "network-metrics-daemon-mgl8h" (UID: "448841dc-bb08-4989-8180-abe966bcaab4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:51.941361 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:51.941263    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:51.941361 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:51.941321    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret podName:319a92b5-37b5-40b2-88cd-5550aab72650 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:55.941305666 +0000 UTC m=+9.069857532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret") pod "global-pull-secret-syncer-vj6pd" (UID: "319a92b5-37b5-40b2-88cd-5550aab72650") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:52.043240 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:52.042109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xff\" (UniqueName: \"kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff\") pod \"network-check-target-m4rw5\" (UID: \"8f387957-9771-467e-9053-994d10fc652a\") " pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:52.048225 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:52.043630    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:30:52.048225 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:52.043659    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:30:52.048225 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:52.043680    2576 projected.go:196] Error preparing data for projected volume kube-api-access-w6xff for pod openshift-network-diagnostics/network-check-target-m4rw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:52.048225 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:52.044294    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff podName:8f387957-9771-467e-9053-994d10fc652a nodeName:}" failed. No retries permitted until 2026-11-03 19:30:56.04426266 +0000 UTC m=+9.172814519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6xff" (UniqueName: "kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff") pod "network-check-target-m4rw5" (UID: "8f387957-9771-467e-9053-994d10fc652a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:52.364518 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:52.363313    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:52.364518 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:52.363444    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:30:52.364518 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:52.363819    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:52.364518 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:52.363910    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:30:52.364518 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:52.363969    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:52.364518 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:52.364046    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:30:53.456125 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:53.455791    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4jl9f" event={"ID":"0a4d21a2-f237-4ffb-88c9-1855e8b4b887","Type":"ContainerStarted","Data":"d53fadedae04699d223243af026896d6586671719a4f6d224f19a40e78ad6baa"}
Nov 03 19:30:53.460871 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:53.460843    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal" event={"ID":"d9cc75c72d3d8481cfe8602c9dd164cf","Type":"ContainerStarted","Data":"ff892b49bec1886c11603ca64f378e4114e2daaa32e15f174fc41c35c00757ed"}
Nov 03 19:30:53.464457 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:53.464424    2576 generic.go:356] "Generic (PLEG): container finished" podID="50b103dd723deeb9c86a7ed93f39fb9c" containerID="01201bacd0aeab71e6c83a8e706ba7fd408437df986516d3212535780305ede3" exitCode=0
Nov 03 19:30:53.464591 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:53.464509    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal" event={"ID":"50b103dd723deeb9c86a7ed93f39fb9c","Type":"ContainerDied","Data":"01201bacd0aeab71e6c83a8e706ba7fd408437df986516d3212535780305ede3"}
Nov 03 19:30:53.471851 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:53.471730    2576 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="fda3c7f10fe670f644f02dff5bc95dc8a6a19b8d9fee89abd5f8098dd4d5c97c" exitCode=0
Nov 03 19:30:53.471851 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:53.471840    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2kqz" event={"ID":"ad438ca5-2f83-4e9b-8058-69edd81f888e","Type":"ContainerDied","Data":"fda3c7f10fe670f644f02dff5bc95dc8a6a19b8d9fee89abd5f8098dd4d5c97c"}
Nov 03 19:30:53.483918 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:53.483858    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4jl9f" podStartSLOduration=2.019736861 podStartE2EDuration="6.483840896s" totalImagesPullingTime="4.464104035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:53.470154644 +0000 UTC m=+6.598706520" watchObservedRunningTime="2026-11-03 19:30:53.483840896 +0000 UTC m=+6.612392773"
Nov 03 19:30:53.515725 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:53.515657    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal" podStartSLOduration=6.515637778 podStartE2EDuration="6.515637778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:53.497737192 +0000 UTC m=+6.626289067" watchObservedRunningTime="2026-11-03 19:30:53.515637778 +0000 UTC m=+6.644189653"
Nov 03 19:30:54.363699 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:54.363665    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:54.363895 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:54.363791    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:30:54.364239 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:54.364216    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:54.364369 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:54.364320    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:30:54.364437 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:54.364379    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:54.364488 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:54.364461    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:30:54.478670 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:54.478611    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal" event={"ID":"50b103dd723deeb9c86a7ed93f39fb9c","Type":"ContainerStarted","Data":"eafedaf238ee2922474cafb4a0a240fb18935083b5eae1bbee41c79122ea37e3"}
Nov 03 19:30:55.495347 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:55.494462    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gcvlj" event={"ID":"adabc7a3-99a6-43f0-b1ed-8a0395d310a4","Type":"ContainerStarted","Data":"c00ccd267c52118f0e53104c7242dff8e04b1e817e40d56000f14ec591cb39dc"}
Nov 03 19:30:55.498465 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:55.498436    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2" event={"ID":"44c7af65-aff0-41b1-8f98-2adf11d53954","Type":"ContainerStarted","Data":"6f5935c7ad3fe9ee896572586720f0278b49ea64c015030077e9ed854c67a7ff"}
Nov 03 19:30:55.507631 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:55.507580    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal" podStartSLOduration=8.507563473 podStartE2EDuration="8.507563473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:54.492830145 +0000 UTC m=+7.621382020" watchObservedRunningTime="2026-11-03 19:30:55.507563473 +0000 UTC m=+8.636115344"
Nov 03 19:30:55.508210 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:55.508148    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gcvlj" podStartSLOduration=2.830513343 podStartE2EDuration="8.508140355s" totalImagesPullingTime="5.677627012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:55.507458318 +0000 UTC m=+8.636010195" watchObservedRunningTime="2026-11-03 19:30:55.508140355 +0000 UTC m=+8.636692264"
Nov 03 19:30:55.978777 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:55.978730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:55.978963 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:55.978804    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:55.978963 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:55.978945    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:55.979098 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:55.979007    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs podName:448841dc-bb08-4989-8180-abe966bcaab4 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:03.978988295 +0000 UTC m=+17.107540165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs") pod "network-metrics-daemon-mgl8h" (UID: "448841dc-bb08-4989-8180-abe966bcaab4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:55.979439 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:55.979420    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:55.979540 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:55.979471    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret podName:319a92b5-37b5-40b2-88cd-5550aab72650 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:03.97945618 +0000 UTC m=+17.108008034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret") pod "global-pull-secret-syncer-vj6pd" (UID: "319a92b5-37b5-40b2-88cd-5550aab72650") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:56.036330 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:56.034837    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:30:56.037073 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:56.036892    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:30:56.081054 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:56.080363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xff\" (UniqueName: \"kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff\") pod \"network-check-target-m4rw5\" (UID: \"8f387957-9771-467e-9053-994d10fc652a\") " pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:56.081054 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:56.080542    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:30:56.081054 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:56.080566    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:30:56.081054 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:56.080606    2576 projected.go:196] Error preparing data for projected volume kube-api-access-w6xff for pod openshift-network-diagnostics/network-check-target-m4rw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:56.081054 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:56.080688    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff podName:8f387957-9771-467e-9053-994d10fc652a nodeName:}" failed. No retries permitted until 2026-11-03 19:31:04.080672518 +0000 UTC m=+17.209224371 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6xff" (UniqueName: "kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff") pod "network-check-target-m4rw5" (UID: "8f387957-9771-467e-9053-994d10fc652a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:56.363330 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:56.362907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:56.363330 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:56.363037    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:30:56.363330 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:56.363100    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:56.363330 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:56.363151    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:30:56.363330 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:56.363221    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:56.363330 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:56.363299    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:30:56.658456 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:56.658421    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:30:57.361347 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:57.361242    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:30:56.658447412Z","UUID":"745b2308-5edd-4cec-84fb-37a1d5f5b01d","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:30:57.362881 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:57.362856    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:30:57.363025 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:57.362887    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:30:57.506366 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:57.506287    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vbrfn" event={"ID":"1697fe5f-c2b8-46e3-b57d-73afd75e6926","Type":"ContainerStarted","Data":"6ed481ab8409c830934722a782088f5f3f1a5a2d3ba53d9e0e727303f8ba4801"}
Nov 03 19:30:57.513910 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:57.513874    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2" event={"ID":"44c7af65-aff0-41b1-8f98-2adf11d53954","Type":"ContainerStarted","Data":"930b456523335455df83ee0e0ad90c0c11a778facedbbcff3beadf8a01852264"}
Nov 03 19:30:57.516225 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:57.516192    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q22m7" event={"ID":"8c63d965-5a4f-481a-8b25-0ac731629615","Type":"ContainerStarted","Data":"f406d8e90e2f6c3f83b30d5af01ff3443d9f44f273b36156225ab4daff626009"}
Nov 03 19:30:57.532761 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:57.532696    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vbrfn" podStartSLOduration=2.288639602 podStartE2EDuration="10.532675312s" totalImagesPullingTime="8.24403571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:57.521296967 +0000 UTC m=+10.649848841" watchObservedRunningTime="2026-11-03 19:30:57.532675312 +0000 UTC m=+10.661227189"
Nov 03 19:30:58.363309 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:58.363272    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:30:58.363779 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:58.363396    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:30:58.363779 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:58.363291    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:30:58.363779 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:58.363470    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:30:58.363779 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:58.363280    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:30:58.363779 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:30:58.363523    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:30:58.521291 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:58.521252    2576 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="dffcf25959aea887b99a007709d78a0998c119cdcc402c25fece6ac7dba10310" exitCode=0
Nov 03 19:30:58.521471 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:58.521341    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2kqz" event={"ID":"ad438ca5-2f83-4e9b-8058-69edd81f888e","Type":"ContainerDied","Data":"dffcf25959aea887b99a007709d78a0998c119cdcc402c25fece6ac7dba10310"}
Nov 03 19:30:58.527663 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:58.527628    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2" event={"ID":"44c7af65-aff0-41b1-8f98-2adf11d53954","Type":"ContainerStarted","Data":"1489e7be75ab6028ba5b2702b9735958db0195fa933d92ee149860134690b8e7"}
Nov 03 19:30:58.538687 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:58.538630    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q22m7" podStartSLOduration=3.083656945 podStartE2EDuration="11.538613164s" totalImagesPullingTime="8.454956219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:57.534002555 +0000 UTC m=+10.662554430" watchObservedRunningTime="2026-11-03 19:30:58.538613164 +0000 UTC m=+11.667165038"
Nov 03 19:30:59.531057 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:59.530721    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mfjxm" event={"ID":"c455037a-fc47-4c3d-a917-854f924a5df0","Type":"ContainerStarted","Data":"afe0c4888dc8444379333f43de07f5affd60a27865bdcd8d366ea005322b5609"}
Nov 03 19:30:59.533341 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:59.533313    2576 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="3ef77b8f45a694988c6e79088dd00cb5013eb47dd099d95a1aedf3a956d34837" exitCode=0
Nov 03 19:30:59.533466 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:59.533391    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2kqz" event={"ID":"ad438ca5-2f83-4e9b-8058-69edd81f888e","Type":"ContainerDied","Data":"3ef77b8f45a694988c6e79088dd00cb5013eb47dd099d95a1aedf3a956d34837"}
Nov 03 19:30:59.535456 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:59.535426    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qmfcq" event={"ID":"34118633-5be6-49c0-9663-8c66046d847a","Type":"ContainerStarted","Data":"7f35384671739f876a76ed82ed85da992d89a62502cfd49e22596404873b513c"}
Nov 03 19:30:59.545195 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:59.545128    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2" podStartSLOduration=3.858519424 podStartE2EDuration="12.545112437s" totalImagesPullingTime="8.686593013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:58.554987495 +0000 UTC m=+11.683539370" watchObservedRunningTime="2026-11-03 19:30:59.545112437 +0000 UTC m=+12.673664312"
Nov 03 19:30:59.545472 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:59.545438    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mfjxm" podStartSLOduration=2.172951648 podStartE2EDuration="12.545430911s" totalImagesPullingTime="10.372479263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:59.545330656 +0000 UTC m=+12.673882521" watchObservedRunningTime="2026-11-03 19:30:59.545430911 +0000 UTC m=+12.673982786"
Nov 03 19:30:59.557068 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:30:59.557015    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qmfcq" podStartSLOduration=4.185367615 podStartE2EDuration="12.556999865s" totalImagesPullingTime="8.37163225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:59.556605683 +0000 UTC m=+12.685157557" watchObservedRunningTime="2026-11-03 19:30:59.556999865 +0000 UTC m=+12.685551739"
Nov 03 19:31:00.363013 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:00.362968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:00.363013 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:00.362988    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:00.363298 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:00.363035    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:00.363298 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:00.363185    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:31:00.363298 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:00.363275    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:31:00.363454 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:00.363360    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:31:01.547913 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:01.547879    2576 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="ff2d5f56cf4d107ea2800409b3f6be160cd4640e7b67b2daf71cb1fe37a40478" exitCode=0
Nov 03 19:31:01.548678 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:01.547932    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2kqz" event={"ID":"ad438ca5-2f83-4e9b-8058-69edd81f888e","Type":"ContainerDied","Data":"ff2d5f56cf4d107ea2800409b3f6be160cd4640e7b67b2daf71cb1fe37a40478"}
Nov 03 19:31:02.364454 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:02.363936    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:02.364454 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:02.364006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:02.364454 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:02.364031    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:02.364454 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:02.364127    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:31:02.364454 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:02.364190    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:31:02.364454 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:02.364264    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:31:03.975296 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:03.975032    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:31:03.975744 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:03.975434    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:31:03.975744 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:03.975707    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:31:04.054017 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.053982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:04.054104 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.054053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:04.054232 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:04.054214    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:31:04.054303 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:04.054283    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret podName:319a92b5-37b5-40b2-88cd-5550aab72650 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:20.054263564 +0000 UTC m=+33.182815419 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret") pod "global-pull-secret-syncer-vj6pd" (UID: "319a92b5-37b5-40b2-88cd-5550aab72650") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:31:04.054303 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:04.054214    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:31:04.054401 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:04.054338    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs podName:448841dc-bb08-4989-8180-abe966bcaab4 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:20.054327262 +0000 UTC m=+33.182879114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs") pod "network-metrics-daemon-mgl8h" (UID: "448841dc-bb08-4989-8180-abe966bcaab4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:31:04.155227 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.155203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xff\" (UniqueName: \"kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff\") pod \"network-check-target-m4rw5\" (UID: \"8f387957-9771-467e-9053-994d10fc652a\") " pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:04.155306 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:04.155290    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:31:04.155342 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:04.155313    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:31:04.155342 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:04.155323    2576 projected.go:196] Error preparing data for projected volume kube-api-access-w6xff for pod openshift-network-diagnostics/network-check-target-m4rw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:31:04.155398 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:04.155370    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff podName:8f387957-9771-467e-9053-994d10fc652a nodeName:}" failed. No retries permitted until 2026-11-03 19:31:20.155354755 +0000 UTC m=+33.283906624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6xff" (UniqueName: "kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff") pod "network-check-target-m4rw5" (UID: "8f387957-9771-467e-9053-994d10fc652a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:31:04.363537 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.363309    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:04.363699 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.363308    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:04.363699 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:04.363572    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:31:04.363699 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:04.363674    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:31:04.363805 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.363328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:04.363805 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:04.363797    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:31:04.557392 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.557366    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:31:04.557711 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.557691    2576 generic.go:356] "Generic (PLEG): container finished" podID="4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1" containerID="8401c705e45efa0503aacd9e2c77babf4de46b83f7a64a6917da1474d50b77d1" exitCode=1
Nov 03 19:31:04.557801 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.557778    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h" event={"ID":"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1","Type":"ContainerStarted","Data":"85fb889affe84bcc299cfe592a5108bf49950e23a7c06ac23aff3d9ddd003086"}
Nov 03 19:31:04.557863 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.557852    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h" event={"ID":"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1","Type":"ContainerStarted","Data":"dc2d70ea5b28f503b75547d3ba12b583b146bf040733bb9e6dae62bb948cf8f3"}
Nov 03 19:31:04.557904 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.557887    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h" event={"ID":"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1","Type":"ContainerStarted","Data":"f73310b0d3e0cd1b6d46ed6e917a561db0967cb9cb39a1ccc79a2f5d948c5647"}
Nov 03 19:31:04.557932 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.557917    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h" event={"ID":"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1","Type":"ContainerStarted","Data":"86bf72f727293b6c9fd0f2bd357649317747f4483fe63a75ded60ee97dc54509"}
Nov 03 19:31:04.557962 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.557948    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h" event={"ID":"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1","Type":"ContainerDied","Data":"8401c705e45efa0503aacd9e2c77babf4de46b83f7a64a6917da1474d50b77d1"}
Nov 03 19:31:04.558019 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.557978    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h" event={"ID":"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1","Type":"ContainerStarted","Data":"8bbfa08ff2c113b3c5b8eba5cc76b7d9bd7685a9df88abccbfa5da680cadcd77"}
Nov 03 19:31:04.560546 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.560514    2576 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="75deb1e18366bed76ad76d48dde808b000899d3cc037c3e9527038bc8837714d" exitCode=0
Nov 03 19:31:04.560655 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:04.560589    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2kqz" event={"ID":"ad438ca5-2f83-4e9b-8058-69edd81f888e","Type":"ContainerDied","Data":"75deb1e18366bed76ad76d48dde808b000899d3cc037c3e9527038bc8837714d"}
Nov 03 19:31:05.566079 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:05.566045    2576 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="e03be54ddbfa0bd66c0a9fd205ef44d62e1cd21fdb1508d48323ed4f50372b99" exitCode=0
Nov 03 19:31:05.566711 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:05.566105    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2kqz" event={"ID":"ad438ca5-2f83-4e9b-8058-69edd81f888e","Type":"ContainerDied","Data":"e03be54ddbfa0bd66c0a9fd205ef44d62e1cd21fdb1508d48323ed4f50372b99"}
Nov 03 19:31:06.363372 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:06.363340    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:06.363372 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:06.363366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:06.363555 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:06.363527    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:31:06.363599 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:06.363570    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:06.363668 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:06.363651    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:31:06.363750 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:06.363731    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:31:06.571805 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:06.571718    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:31:06.572237 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:06.572149    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h" event={"ID":"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1","Type":"ContainerStarted","Data":"46284092c3416127a2e8a0d8cbf26ef1a3e52b9cce696ba10c38f403a343ec15"}
Nov 03 19:31:06.575361 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:06.575334    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2kqz" event={"ID":"ad438ca5-2f83-4e9b-8058-69edd81f888e","Type":"ContainerStarted","Data":"e0d8b1a0b3fea6c9aa6e45bef2a508d7d7d8607ba565416544172dea3c5aa426"}
Nov 03 19:31:06.592608 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:06.592561    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w2kqz" podStartSLOduration=8.616048641 podStartE2EDuration="19.592549593s" totalImagesPullingTime="10.976500952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:06.591539368 +0000 UTC m=+19.720091242" watchObservedRunningTime="2026-11-03 19:31:06.592549593 +0000 UTC m=+19.721101467"
Nov 03 19:31:08.363404 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:08.363370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:08.363979 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:08.363375    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:08.363979 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:08.363515    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:31:08.363979 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:08.363562    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:31:08.363979 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:08.363380    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:08.363979 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:08.363674    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:31:08.583040 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:08.583013    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:31:08.583373 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:08.583347    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h" event={"ID":"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1","Type":"ContainerStarted","Data":"f2fd15b379401b4044129f5497f694f312e4132c6105321bcc95834439aebfb6"}
Nov 03 19:31:08.583712 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:08.583692    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:08.583797 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:08.583757    2576 scope.go:122] "RemoveContainer" containerID="8401c705e45efa0503aacd9e2c77babf4de46b83f7a64a6917da1474d50b77d1"
Nov 03 19:31:08.583797 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:08.583774    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:08.583915 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:08.583822    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:08.599924 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:08.599903    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:08.600687 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:08.600667    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:09.589354 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:09.589158    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:31:09.590074 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:09.589641    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h" event={"ID":"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1","Type":"ContainerStarted","Data":"9f57d114e9c3f523dcdeb232247d2ad160e9582b7ad7f36f1ecc2fdfb4390772"}
Nov 03 19:31:09.612633 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:09.612569    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h" podStartSLOduration=7.435811779 podStartE2EDuration="22.612555997s" totalImagesPullingTime="15.176744218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:09.611467184 +0000 UTC m=+22.740019080" watchObservedRunningTime="2026-11-03 19:31:09.612555997 +0000 UTC m=+22.741107896"
Nov 03 19:31:10.039872 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:10.039844    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vj6pd"]
Nov 03 19:31:10.040085 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:10.039978    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:10.040159 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:10.040087    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:31:10.040995 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:10.040969    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m4rw5"]
Nov 03 19:31:10.041108 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:10.041078    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:10.041227 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:10.041199    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:31:10.041939 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:10.041721    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgl8h"]
Nov 03 19:31:10.041939 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:10.041808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:10.041939 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:10.041909    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:31:11.363014 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:11.362967    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:11.363014 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:11.363011    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:11.363549 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:11.363100    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:31:11.363549 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:11.363247    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:31:12.363365 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:12.363332    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:12.363780 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:12.363465    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:31:13.363719 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:13.363630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:13.363719 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:13.363642    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:13.364288 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:13.363731    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:31:13.364288 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:13.363798    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:31:14.213528 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:14.213491    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:14.363634 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:14.363603    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:14.363790 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:14.363749    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:31:14.392494 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:14.392465    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:15.363523 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:15.363482    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:15.363704 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:15.363494    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:15.363704 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:15.363595    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m4rw5" podUID="8f387957-9771-467e-9053-994d10fc652a"
Nov 03 19:31:15.363828 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:15.363709    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vj6pd" podUID="319a92b5-37b5-40b2-88cd-5550aab72650"
Nov 03 19:31:16.363589 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.363549    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:16.363766 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:16.363685    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgl8h" podUID="448841dc-bb08-4989-8180-abe966bcaab4"
Nov 03 19:31:16.562019 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.561933    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:16.689389 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.689352    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:16.689693 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.689676    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeReady"
Nov 03 19:31:16.689818 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.689808    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 19:31:16.727536 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.727498    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8pptd"]
Nov 03 19:31:16.731008 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.730985    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:16.731153 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.731108    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tzpld"]
Nov 03 19:31:16.735212 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.735193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.742238 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.742188    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-brhwf\""
Nov 03 19:31:16.742238 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.742199    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:31:16.742238 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.742218    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:31:16.742436 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.742323    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:31:16.742436 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.742338    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:31:16.742599 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.742570    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:31:16.744346 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.742638    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-skb8s\""
Nov 03 19:31:16.744346 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.743504    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8pptd"]
Nov 03 19:31:16.745305 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.745282    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tzpld"]
Nov 03 19:31:16.837388 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.837300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p9qg\" (UniqueName: \"kubernetes.io/projected/99d9bea7-e81a-4fd7-8757-1eadfe124622-kube-api-access-2p9qg\") pod \"ingress-canary-8pptd\" (UID: \"99d9bea7-e81a-4fd7-8757-1eadfe124622\") " pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:16.837388 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.837334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99d9bea7-e81a-4fd7-8757-1eadfe124622-cert\") pod \"ingress-canary-8pptd\" (UID: \"99d9bea7-e81a-4fd7-8757-1eadfe124622\") " pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:16.837388 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.837352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4473c0-57b5-44a8-a467-d6a29d958c68-config-volume\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.837611 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.837457    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4473c0-57b5-44a8-a467-d6a29d958c68-tmp-dir\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.837611 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.837476    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.837611 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.837510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlmm\" (UniqueName: \"kubernetes.io/projected/1d4473c0-57b5-44a8-a467-d6a29d958c68-kube-api-access-jzlmm\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.938697 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.938658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlmm\" (UniqueName: \"kubernetes.io/projected/1d4473c0-57b5-44a8-a467-d6a29d958c68-kube-api-access-jzlmm\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.938697 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.938701    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p9qg\" (UniqueName: \"kubernetes.io/projected/99d9bea7-e81a-4fd7-8757-1eadfe124622-kube-api-access-2p9qg\") pod \"ingress-canary-8pptd\" (UID: \"99d9bea7-e81a-4fd7-8757-1eadfe124622\") " pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:16.938899 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.938718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99d9bea7-e81a-4fd7-8757-1eadfe124622-cert\") pod \"ingress-canary-8pptd\" (UID: \"99d9bea7-e81a-4fd7-8757-1eadfe124622\") " pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:16.938899 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.938743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4473c0-57b5-44a8-a467-d6a29d958c68-config-volume\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.938899 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.938871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4473c0-57b5-44a8-a467-d6a29d958c68-tmp-dir\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.939010 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.938911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.939052 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:16.939017    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:16.939142 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:16.939127    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls podName:1d4473c0-57b5-44a8-a467-d6a29d958c68 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.439106738 +0000 UTC m=+30.567658605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls") pod "dns-default-tzpld" (UID: "1d4473c0-57b5-44a8-a467-d6a29d958c68") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:16.939439 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.939413    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4473c0-57b5-44a8-a467-d6a29d958c68-tmp-dir\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.939679 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.939656    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4473c0-57b5-44a8-a467-d6a29d958c68-config-volume\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.942916 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.942898    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99d9bea7-e81a-4fd7-8757-1eadfe124622-cert\") pod \"ingress-canary-8pptd\" (UID: \"99d9bea7-e81a-4fd7-8757-1eadfe124622\") " pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:16.948774 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.948753    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlmm\" (UniqueName: \"kubernetes.io/projected/1d4473c0-57b5-44a8-a467-d6a29d958c68-kube-api-access-jzlmm\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:16.948888 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:16.948870    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p9qg\" (UniqueName: \"kubernetes.io/projected/99d9bea7-e81a-4fd7-8757-1eadfe124622-kube-api-access-2p9qg\") pod \"ingress-canary-8pptd\" (UID: \"99d9bea7-e81a-4fd7-8757-1eadfe124622\") " pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:17.044926 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.044881    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:17.207237 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.207207    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:17.325596 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.325564    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8pptd"]
Nov 03 19:31:17.328816 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:31:17.328786    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d9bea7_e81a_4fd7_8757_1eadfe124622.slice/crio-40e32d134a03bfbfb2eb354c4a071facdc1ce0bb0ca0c2b301167a47dbc12278 WatchSource:0}: Error finding container 40e32d134a03bfbfb2eb354c4a071facdc1ce0bb0ca0c2b301167a47dbc12278: Status 404 returned error can't find the container with id 40e32d134a03bfbfb2eb354c4a071facdc1ce0bb0ca0c2b301167a47dbc12278
Nov 03 19:31:17.329820 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.329796    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:17.364001 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.363934    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:17.364129 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.364099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:17.368212 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.368187    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:31:17.368388 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.368370    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:31:17.368388 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.368380    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-77jlf\""
Nov 03 19:31:17.368510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.368463    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khl78\""
Nov 03 19:31:17.368510 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.368470    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:31:17.442216 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.442163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:17.442365 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:17.442306    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:17.442416 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:17.442366    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls podName:1d4473c0-57b5-44a8-a467-d6a29d958c68 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:18.442349643 +0000 UTC m=+31.570901499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls") pod "dns-default-tzpld" (UID: "1d4473c0-57b5-44a8-a467-d6a29d958c68") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:17.612053 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:17.612018    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8pptd" event={"ID":"99d9bea7-e81a-4fd7-8757-1eadfe124622","Type":"ContainerStarted","Data":"40e32d134a03bfbfb2eb354c4a071facdc1ce0bb0ca0c2b301167a47dbc12278"}
Nov 03 19:31:18.363835 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:18.363794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:18.366332 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:18.366309    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:31:18.366882 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:18.366864    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tdq2h\""
Nov 03 19:31:18.450240 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:18.450203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:18.450440 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:18.450349    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:18.450440 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:18.450399    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls podName:1d4473c0-57b5-44a8-a467-d6a29d958c68 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:20.450385574 +0000 UTC m=+33.578937439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls") pod "dns-default-tzpld" (UID: "1d4473c0-57b5-44a8-a467-d6a29d958c68") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:18.958875 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:18.958828    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4"]
Nov 03 19:31:18.982303 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:18.982272    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4"]
Nov 03 19:31:18.982490 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:18.982408    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4"
Nov 03 19:31:18.985022 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:18.984934    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 19:31:18.985321 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:18.985280    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:18.985516 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:18.985486    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2lfv2\""
Nov 03 19:31:18.993217 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:18.993188    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:19.130124 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:19.130088    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:19.155100 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:19.155061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz5nb\" (UniqueName: \"kubernetes.io/projected/ab89d762-778b-4e85-b506-72be48727030-kube-api-access-hz5nb\") pod \"migrator-66df5d496b-fk9b4\" (UID: \"ab89d762-778b-4e85-b506-72be48727030\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4"
Nov 03 19:31:19.248831 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:19.248739    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:19.256392 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:19.256354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5nb\" (UniqueName: \"kubernetes.io/projected/ab89d762-778b-4e85-b506-72be48727030-kube-api-access-hz5nb\") pod \"migrator-66df5d496b-fk9b4\" (UID: \"ab89d762-778b-4e85-b506-72be48727030\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4"
Nov 03 19:31:19.263686 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:19.263659    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5nb\" (UniqueName: \"kubernetes.io/projected/ab89d762-778b-4e85-b506-72be48727030-kube-api-access-hz5nb\") pod \"migrator-66df5d496b-fk9b4\" (UID: \"ab89d762-778b-4e85-b506-72be48727030\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4"
Nov 03 19:31:19.306730 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:19.306693    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4"
Nov 03 19:31:19.598578 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:19.598497    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4"]
Nov 03 19:31:19.603332 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:31:19.603302    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab89d762_778b_4e85_b506_72be48727030.slice/crio-42ca6cbb98a2c823c5bae1cefe5456497ce81993b81d2184264e7c58e99f4783 WatchSource:0}: Error finding container 42ca6cbb98a2c823c5bae1cefe5456497ce81993b81d2184264e7c58e99f4783: Status 404 returned error can't find the container with id 42ca6cbb98a2c823c5bae1cefe5456497ce81993b81d2184264e7c58e99f4783
Nov 03 19:31:19.617588 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:19.617556    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4" event={"ID":"ab89d762-778b-4e85-b506-72be48727030","Type":"ContainerStarted","Data":"42ca6cbb98a2c823c5bae1cefe5456497ce81993b81d2184264e7c58e99f4783"}
Nov 03 19:31:20.062648 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.062612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:20.063058 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.062693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:20.063058 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:20.062803    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 19:31:20.063058 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:20.062878    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs podName:448841dc-bb08-4989-8180-abe966bcaab4 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:52.062858791 +0000 UTC m=+65.191410653 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs") pod "network-metrics-daemon-mgl8h" (UID: "448841dc-bb08-4989-8180-abe966bcaab4") : secret "metrics-daemon-secret" not found
Nov 03 19:31:20.064993 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.064974    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:20.082695 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.082663    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:20.163030 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.162989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xff\" (UniqueName: \"kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff\") pod \"network-check-target-m4rw5\" (UID: \"8f387957-9771-467e-9053-994d10fc652a\") " pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:20.165505 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.165487    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6xff\" (UniqueName: \"kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff\") pod \"network-check-target-m4rw5\" (UID: \"8f387957-9771-467e-9053-994d10fc652a\") " pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:20.384195 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.384055    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:20.448126 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.448095    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vj6pd"]
Nov 03 19:31:20.450281 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.450219    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:20.465685 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.465659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:20.465797 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:20.465767    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:20.465860 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:20.465812    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls podName:1d4473c0-57b5-44a8-a467-d6a29d958c68 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:24.465798289 +0000 UTC m=+37.594350141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls") pod "dns-default-tzpld" (UID: "1d4473c0-57b5-44a8-a467-d6a29d958c68") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:20.591118 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.591082    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:20.621359 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.621321    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8pptd" event={"ID":"99d9bea7-e81a-4fd7-8757-1eadfe124622","Type":"ContainerStarted","Data":"94a7e2ff7b01a2ec2348ec56b97fefd6ee78b08c2ac082816435254cb1044d60"}
Nov 03 19:31:20.622596 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.622568    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vj6pd" event={"ID":"319a92b5-37b5-40b2-88cd-5550aab72650","Type":"ContainerStarted","Data":"dff09f0062277125e74fe5e5fb9224577cbceb7a1d19a7a3c8e164d85ab7ddf5"}
Nov 03 19:31:20.637096 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.637007    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8pptd" podStartSLOduration=1.777517258 podStartE2EDuration="4.636988521s" totalImagesPullingTime="2.859471263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:20.635515014 +0000 UTC m=+33.764066889" watchObservedRunningTime="2026-11-03 19:31:20.636988521 +0000 UTC m=+33.765540401"
Nov 03 19:31:20.650427 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.650391    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pdfb6"]
Nov 03 19:31:20.666937 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.666308    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pdfb6"]
Nov 03 19:31:20.666937 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.666375    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m4rw5"]
Nov 03 19:31:20.666937 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.666511    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:20.669505 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:31:20.669473    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f387957_9771_467e_9053_994d10fc652a.slice/crio-db03ac7844ab2ae3c86f30aa6077e8853ff741a9d59ffb08f9c919d9add35679 WatchSource:0}: Error finding container db03ac7844ab2ae3c86f30aa6077e8853ff741a9d59ffb08f9c919d9add35679: Status 404 returned error can't find the container with id db03ac7844ab2ae3c86f30aa6077e8853ff741a9d59ffb08f9c919d9add35679
Nov 03 19:31:20.670907 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.670878    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 19:31:20.674975 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.674936    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gbtlj\""
Nov 03 19:31:20.680061 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.680026    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 19:31:20.680157 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.680085    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 19:31:20.680246 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.680150    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 19:31:20.749185 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.749138    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:20.769534 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.769499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwvm6\" (UniqueName: \"kubernetes.io/projected/bbbb1119-eb4e-48f1-8d3c-2c9264080717-kube-api-access-nwvm6\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:20.769691 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.769546    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-key\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:20.769762 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.769738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-cabundle\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:20.776191 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.776149    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-9xxl7-qt48t"]
Nov 03 19:31:20.782128 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.782108    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rjzpc"]
Nov 03 19:31:20.782270 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.782250    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.787957 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.787929    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9xxl7-qt48t"]
Nov 03 19:31:20.788218 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.788049    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.788840 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.788820    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:31:20.788840 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.788859    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rjzpc"]
Nov 03 19:31:20.789221 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.789160    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 19:31:20.789426 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.789411    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wrft7\""
Nov 03 19:31:20.789505 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.789494    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:31:20.789950 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.789930    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:31:20.790241 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.790046    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 19:31:20.791764 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.790611    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:31:20.791764 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.790703    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nblsj\""
Nov 03 19:31:20.870911 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.870874    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-trusted-ca-bundle\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.870911 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.870919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xng5d\" (UniqueName: \"kubernetes.io/projected/ba2d3316-f9d0-4812-89a3-d34efcc04628-kube-api-access-xng5d\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.871337 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.870953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-cabundle\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:20.871337 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.871073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.871337 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.871112    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.871337 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.871160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba2d3316-f9d0-4812-89a3-d34efcc04628-archives-path\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.871337 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.871292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwvm6\" (UniqueName: \"kubernetes.io/projected/bbbb1119-eb4e-48f1-8d3c-2c9264080717-kube-api-access-nwvm6\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:20.871337 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.871333    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx2mr\" (UniqueName: \"kubernetes.io/projected/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-api-access-lx2mr\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.871655 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.871390    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9abeb54a-e625-4ad1-827e-49f5640d6288-data-volume\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.871655 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.871420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-key\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:20.871655 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.871445    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9abeb54a-e625-4ad1-827e-49f5640d6288-crio-socket\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.871655 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.871469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-service-ca-bundle\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.871826 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.871681    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-cabundle\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:20.874369 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.874345    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-key\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:20.879347 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.879317    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwvm6\" (UniqueName: \"kubernetes.io/projected/bbbb1119-eb4e-48f1-8d3c-2c9264080717-kube-api-access-nwvm6\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:20.881254 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.881229    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:20.972806 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.972714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx2mr\" (UniqueName: \"kubernetes.io/projected/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-api-access-lx2mr\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.972806 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.972779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9abeb54a-e625-4ad1-827e-49f5640d6288-data-volume\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.973032 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.972970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9abeb54a-e625-4ad1-827e-49f5640d6288-crio-socket\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.973032 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.973016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-service-ca-bundle\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.973140 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.973058    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-trusted-ca-bundle\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.973140 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.973091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng5d\" (UniqueName: \"kubernetes.io/projected/ba2d3316-f9d0-4812-89a3-d34efcc04628-kube-api-access-xng5d\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.973140 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.973102    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9abeb54a-e625-4ad1-827e-49f5640d6288-data-volume\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.973140 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.973126    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.973354 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.973159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.973354 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.973213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba2d3316-f9d0-4812-89a3-d34efcc04628-archives-path\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.973354 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.973213    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9abeb54a-e625-4ad1-827e-49f5640d6288-crio-socket\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.973354 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:20.973242    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:20.973354 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:20.973266    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 19:31:20.973354 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:20.973328    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm podName:9abeb54a-e625-4ad1-827e-49f5640d6288 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.473310357 +0000 UTC m=+34.601862226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjzpc" (UID: "9abeb54a-e625-4ad1-827e-49f5640d6288") : configmap "kube-rbac-proxy" not found
Nov 03 19:31:20.973354 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:20.973341    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls podName:9abeb54a-e625-4ad1-827e-49f5640d6288 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.473335191 +0000 UTC m=+34.601887043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjzpc" (UID: "9abeb54a-e625-4ad1-827e-49f5640d6288") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:20.973693 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.973583    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba2d3316-f9d0-4812-89a3-d34efcc04628-archives-path\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.973749 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.973693    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-service-ca-bundle\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.973981 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.973962    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-trusted-ca-bundle\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:20.981395 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.981365    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx2mr\" (UniqueName: \"kubernetes.io/projected/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-api-access-lx2mr\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:20.981906 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:20.981883    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xng5d\" (UniqueName: \"kubernetes.io/projected/ba2d3316-f9d0-4812-89a3-d34efcc04628-kube-api-access-xng5d\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:21.005921 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.005892    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:21.007899 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.007870    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:21.122753 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.122721    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:21.301255 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.301032    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pdfb6"]
Nov 03 19:31:21.334391 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:31:21.334354    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbb1119_eb4e_48f1_8d3c_2c9264080717.slice/crio-96e7e5a9ca903443b08336a03d0fc43666a20f0c2426db9f12505183d30acf0a WatchSource:0}: Error finding container 96e7e5a9ca903443b08336a03d0fc43666a20f0c2426db9f12505183d30acf0a: Status 404 returned error can't find the container with id 96e7e5a9ca903443b08336a03d0fc43666a20f0c2426db9f12505183d30acf0a
Nov 03 19:31:21.412161 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.412131    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9xxl7-qt48t"]
Nov 03 19:31:21.422470 ip-10-0-131-234 kubenswrapper[2576]: W1103 19:31:21.422439    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2d3316_f9d0_4812_89a3_d34efcc04628.slice/crio-16e2877f87a0befb90ed4eaf6833e145f834da44968f3b1c2bea606ef7a08fc5 WatchSource:0}: Error finding container 16e2877f87a0befb90ed4eaf6833e145f834da44968f3b1c2bea606ef7a08fc5: Status 404 returned error can't find the container with id 16e2877f87a0befb90ed4eaf6833e145f834da44968f3b1c2bea606ef7a08fc5
Nov 03 19:31:21.478501 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.478467    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:21.478643 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.478515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:21.478762 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:21.478663    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 19:31:21.478762 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:21.478728    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm podName:9abeb54a-e625-4ad1-827e-49f5640d6288 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:22.478706979 +0000 UTC m=+35.607258847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjzpc" (UID: "9abeb54a-e625-4ad1-827e-49f5640d6288") : configmap "kube-rbac-proxy" not found
Nov 03 19:31:21.479204 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:21.479167    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:21.479328 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:21.479237    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls podName:9abeb54a-e625-4ad1-827e-49f5640d6288 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:22.479221615 +0000 UTC m=+35.607773471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjzpc" (UID: "9abeb54a-e625-4ad1-827e-49f5640d6288") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:21.608201 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.608061    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:21.627640 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.627598    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m4rw5" event={"ID":"8f387957-9771-467e-9053-994d10fc652a","Type":"ContainerStarted","Data":"db03ac7844ab2ae3c86f30aa6077e8853ff741a9d59ffb08f9c919d9add35679"}
Nov 03 19:31:21.629545 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.629516    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9xxl7-qt48t" event={"ID":"ba2d3316-f9d0-4812-89a3-d34efcc04628","Type":"ContainerStarted","Data":"16e2877f87a0befb90ed4eaf6833e145f834da44968f3b1c2bea606ef7a08fc5"}
Nov 03 19:31:21.633351 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.633323    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4" event={"ID":"ab89d762-778b-4e85-b506-72be48727030","Type":"ContainerStarted","Data":"a4378bf470336bb219b86332752904f247a14ef4c51e748126feb3166d12285b"}
Nov 03 19:31:21.635479 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.635448    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6" event={"ID":"bbbb1119-eb4e-48f1-8d3c-2c9264080717","Type":"ContainerStarted","Data":"96e7e5a9ca903443b08336a03d0fc43666a20f0c2426db9f12505183d30acf0a"}
Nov 03 19:31:21.649519 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:21.649457    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4" podStartSLOduration=1.889692042 podStartE2EDuration="3.649440482s" totalImagesPullingTime="1.75974844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:21.649412901 +0000 UTC m=+34.777964774" watchObservedRunningTime="2026-11-03 19:31:21.649440482 +0000 UTC m=+34.777992359"
Nov 03 19:31:22.363104 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:22.362998    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:22.487103 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:22.487064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:22.487277 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:22.487121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:22.487360 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:22.487275    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 19:31:22.487360 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:22.487339    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm podName:9abeb54a-e625-4ad1-827e-49f5640d6288 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:24.487322643 +0000 UTC m=+37.615874499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjzpc" (UID: "9abeb54a-e625-4ad1-827e-49f5640d6288") : configmap "kube-rbac-proxy" not found
Nov 03 19:31:22.487781 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:22.487596    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:22.487781 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:22.487716    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls podName:9abeb54a-e625-4ad1-827e-49f5640d6288 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:24.487697634 +0000 UTC m=+37.616249499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjzpc" (UID: "9abeb54a-e625-4ad1-827e-49f5640d6288") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:22.492286 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:22.492218    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:24.506618 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:24.506189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:24.506618 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:24.506279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:24.506618 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:24.506293    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:24.506618 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:24.506376    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 19:31:24.506618 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:24.506489    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:24.506618 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:24.506379    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls podName:1d4473c0-57b5-44a8-a467-d6a29d958c68 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:32.506358345 +0000 UTC m=+45.634910224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls") pod "dns-default-tzpld" (UID: "1d4473c0-57b5-44a8-a467-d6a29d958c68") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:24.506618 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:24.506536    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm podName:9abeb54a-e625-4ad1-827e-49f5640d6288 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.506519498 +0000 UTC m=+41.635071365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjzpc" (UID: "9abeb54a-e625-4ad1-827e-49f5640d6288") : configmap "kube-rbac-proxy" not found
Nov 03 19:31:24.506618 ip-10-0-131-234 kubenswrapper[2576]: E1103 19:31:24.506552    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls podName:9abeb54a-e625-4ad1-827e-49f5640d6288 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.506542465 +0000 UTC m=+41.635094330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjzpc" (UID: "9abeb54a-e625-4ad1-827e-49f5640d6288") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:24.506618 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:24.506312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:24.802317 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:24.802235    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:24.989637 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:24.989600    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:25.650633 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:25.649822    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6" event={"ID":"bbbb1119-eb4e-48f1-8d3c-2c9264080717","Type":"ContainerStarted","Data":"6980e816a04dce43eab486f1da9dbf3d0c06ebd5bd31dbebdd769911fdc6534e"}
Nov 03 19:31:25.654255 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:25.653921    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m4rw5" event={"ID":"8f387957-9771-467e-9053-994d10fc652a","Type":"ContainerStarted","Data":"84eefdeba3f0d0a9f57906b004955ea9148e6da47eaca876440cabb2daaf5dc4"}
Nov 03 19:31:25.654255 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:25.654075    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:25.655430 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:25.655396    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9xxl7-qt48t" event={"ID":"ba2d3316-f9d0-4812-89a3-d34efcc04628","Type":"ContainerStarted","Data":"db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c"}
Nov 03 19:31:25.664954 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:25.664906    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6" podStartSLOduration=2.143904916 podStartE2EDuration="5.664888887s" totalImagesPullingTime="3.520983971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:25.664370849 +0000 UTC m=+38.792922721" watchObservedRunningTime="2026-11-03 19:31:25.664888887 +0000 UTC m=+38.793440758"
Nov 03 19:31:25.698752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:25.698388    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-9xxl7-qt48t" podStartSLOduration=2.257526502 podStartE2EDuration="5.698375222s" totalImagesPullingTime="3.44084872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:25.681818056 +0000 UTC m=+38.810369933" watchObservedRunningTime="2026-11-03 19:31:25.698375222 +0000 UTC m=+38.826927099"
Nov 03 19:31:25.698752 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:25.698558    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m4rw5" podStartSLOduration=34.151737683 podStartE2EDuration="38.69855183s" totalImagesPullingTime="4.546814147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:25.696200266 +0000 UTC m=+38.824752138" watchObservedRunningTime="2026-11-03 19:31:25.69855183 +0000 UTC m=+38.827103705"
Nov 03 19:31:25.734229 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:25.734150    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:25.861597 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:25.861563    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:26.000811 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:26.000735    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:26.412432 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:26.412397    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:26.659658 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:26.659610    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vj6pd" event={"ID":"319a92b5-37b5-40b2-88cd-5550aab72650","Type":"ContainerStarted","Data":"e205f2c52f56263a9f4b58dc743354387ffc2525daa61fb61f272c0d5853908e"}
Nov 03 19:31:26.672537 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:26.672493    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vj6pd" podStartSLOduration=33.756026145999996 podStartE2EDuration="39.672480349s" totalImagesPullingTime="5.916454203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:26.672003199 +0000 UTC m=+39.800555099" watchObservedRunningTime="2026-11-03 19:31:26.672480349 +0000 UTC m=+39.801032234"
Nov 03 19:31:26.677358 ip-10-0-131-234 kubenswrapper[2576]: I1103 19:31:26.677281    2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:31:26.677419 ip-10-0-131-234 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 19:31:26.684606 ip-10-0-131-234 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 19:31:26.684782 ip-10-0-131-234 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 19:31:26.684840 ip-10-0-131-234 systemd[1]: kubelet.service: Consumed 2.446s CPU time.
Nov 03 19:31:26.697906 ip-10-0-131-234 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:31:26.775316 ip-10-0-131-234 kubenswrapper[4531]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:31:26.775316 ip-10-0-131-234 kubenswrapper[4531]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:31:26.775316 ip-10-0-131-234 kubenswrapper[4531]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:31:26.775316 ip-10-0-131-234 kubenswrapper[4531]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:31:26.777435 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777347    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.777435 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777435    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777440    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777443    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777446    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777449    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777452    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777464    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777467    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777470    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777472    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777475    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777478    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777480    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777483    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777488    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777492    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777496    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777498    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777502    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777504    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.777511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777507    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777510    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777513    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777516    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777519    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777522    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777525    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777528    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777530    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777533    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777536    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777538    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777541    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777543    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777546    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777548    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777551    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777553    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777557    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.778006 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777560    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777562    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777565    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777567    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777570    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777572    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777575    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777578    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777580    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777583    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777586    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777590    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777593    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777596    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777599    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777601    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777604    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777607    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777609    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.778481 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777612    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777615    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777617    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777620    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777623    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777625    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777628    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777630    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777633    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777636    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777638    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777641    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777644    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777647    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777650    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777652    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777656    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777659    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777662    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777665    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.778953 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777668    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777671    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777674    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777676    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777679    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777682    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777684    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777687    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777690    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777692    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777695    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777697    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777700    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777702    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777705    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777708    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777710    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777712    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777715    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.779508 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777718    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777720    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777722    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777725    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777728    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777730    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777733    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.777736    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778558    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778566    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778569    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778572    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778575    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778579    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778582    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778586    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778589    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778591    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778594    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778597    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.779985 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778600    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778603    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778606    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778608    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778611    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778614    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778616    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778619    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778621    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778624    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778627    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778629    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778632    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778634    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778637    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778639    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778642    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778645    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778647    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778650    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.780485 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778653    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778656    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778659    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778663    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778668    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778672    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778675    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778679    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778682    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778685    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778690    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778700    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778703    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778706    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778709    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778714    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778717    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778719    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778722    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.780977 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778725    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778727    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778730    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778733    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778736    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778738    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778741    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778743    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778746    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778748    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778751    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778753    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778756    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778759    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778765    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778772    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778774    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778777    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778780    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778782    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.781461 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778785    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778787    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778790    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778793    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778797    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778800    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778804    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778808    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778812    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778816    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778820    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778824    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778828    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778831    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778836    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778840    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778843    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778845    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778848    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778851    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.781956 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778854    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778857    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778859    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778863    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778866    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778869    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778872    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778877    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778880    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778883    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778885    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778888    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778891    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778893    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.778896    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779476    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779484    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779487    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779491    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779494    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.782473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779497    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779500    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779503    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779506    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779509    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779512    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779515    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779519    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779523    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779526    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779529    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779532    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779535    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779538    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779540    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779543    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779545    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779548    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779550    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.782954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779554    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779557    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779560    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779562    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779566    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779570    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779574    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779577    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779580    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779583    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779585    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779588    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779591    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779593    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779596    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779599    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779602    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779604    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779607    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.783419 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779609    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779612    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779615    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779617    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779620    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779622    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779625    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779627    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779630    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779633    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779635    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779638    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779640    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779643    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779646    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779649    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779653    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779655    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779658    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779660    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.783897 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779663    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779666    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779671    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779673    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779676    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779679    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779681    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779684    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779687    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779689    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779692    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779695    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779697    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779700    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779702    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779705    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779707    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779710    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779712    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.784526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779715    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779717    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779720    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779723    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779726    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779729    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779731    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779734    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779737    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779739    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779742    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779744    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779747    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779750    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779753    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779756    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779759    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779762    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779764    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779767    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.784987 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779770    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779773    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779775    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.779778    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779868    4531 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779877    4531 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779884    4531 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779888    4531 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779895    4531 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779900    4531 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779906    4531 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779912    4531 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779917    4531 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779920    4531 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779924    4531 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779928    4531 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779931    4531 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779934    4531 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779937    4531 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779940    4531 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779943    4531 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779946    4531 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779951    4531 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779954    4531 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:31:26.785567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779957    4531 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779960    4531 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779964    4531 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779970    4531 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779973    4531 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779976    4531 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779979    4531 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779982    4531 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779985    4531 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779988    4531 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779992    4531 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.779997    4531 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780002    4531 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780005    4531 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780008    4531 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780011    4531 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780014    4531 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780017    4531 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780025    4531 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780029    4531 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780032    4531 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780035    4531 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780038    4531 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780043    4531 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:31:26.786585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780046    4531 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780050    4531 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780053    4531 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780056    4531 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780059    4531 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780061    4531 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780065    4531 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780068    4531 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780071    4531 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780074    4531 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780078    4531 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780083    4531 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780087    4531 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780090    4531 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780093    4531 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780096    4531 flags.go:64] FLAG: --help="false"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780099    4531 flags.go:64] FLAG: --hostname-override="ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780103    4531 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780106    4531 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780111    4531 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780114    4531 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780117    4531 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780120    4531 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:31:26.787278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780123    4531 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780126    4531 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780130    4531 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780133    4531 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780136    4531 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780139    4531 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780142    4531 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780145    4531 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780148    4531 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780151    4531 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780154    4531 flags.go:64] FLAG: --lock-file=""
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780157    4531 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780160    4531 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780163    4531 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780181    4531 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780184    4531 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780188    4531 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780191    4531 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780194    4531 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780198    4531 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780202    4531 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780205    4531 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780209    4531 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780212    4531 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780216    4531 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:31:26.787847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780219    4531 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780222    4531 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780226    4531 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780230    4531 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780233    4531 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780236    4531 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780240    4531 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780251    4531 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780255    4531 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780258    4531 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780261    4531 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780264    4531 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780267    4531 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780270    4531 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780273    4531 flags.go:64] FLAG: --port="10250"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780276    4531 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780279    4531 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0043e197a5031c2a6"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780282    4531 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780286    4531 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780288    4531 flags.go:64] FLAG: --register-node="true"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780291    4531 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780295    4531 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780298    4531 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780301    4531 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780305    4531 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780309    4531 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:31:26.788506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780312    4531 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780317    4531 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780320    4531 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780323    4531 flags.go:64] FLAG: --runonce="false"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780325    4531 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780328    4531 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780332    4531 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780334    4531 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780337    4531 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780342    4531 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780346    4531 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780349    4531 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780352    4531 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780355    4531 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780358    4531 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780361    4531 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780365    4531 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780367    4531 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780370    4531 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780375    4531 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780378    4531 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780382    4531 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780386    4531 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780389    4531 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780391    4531 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:31:26.789139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780394    4531 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780398    4531 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780401    4531 flags.go:64] FLAG: --v="2"
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780406    4531 flags.go:64] FLAG: --version="false"
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780410    4531 flags.go:64] FLAG: --vmodule=""
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780414    4531 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780418    4531 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780559    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780565    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780568    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780571    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780574    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780577    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780580    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780584    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780588    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780593    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780596    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780599    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780602    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.789761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780605    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780608    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780610    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780614    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780616    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780619    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780622    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780624    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780627    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780629    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780632    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780635    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780637    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780640    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780643    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780646    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780648    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780651    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780653    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.790395 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780656    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780660    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780663    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780666    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780668    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780671    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780673    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780676    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780679    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780683    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780686    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780689    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780691    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780694    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780697    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780699    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780702    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780705    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780707    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780710    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.790887 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780712    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780715    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780718    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780720    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780723    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780726    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780728    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780731    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780733    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780736    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780739    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780742    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780744    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780749    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780752    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780754    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780757    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780760    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780762    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780765    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.791578 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780767    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780772    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780774    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780777    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780779    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780782    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780785    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780787    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780790    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780792    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780795    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780798    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780800    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780803    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780806    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780808    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780811    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780813    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780816    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780819    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.792103 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780821    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780825    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780828    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780831    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780834    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780838    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780841    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780844    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780846    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780849    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780851    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780854    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780857    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780861    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.780887    4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:31:26.792938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780921    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780925    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780928    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780931    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780934    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780936    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780939    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780942    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780946    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780949    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780952    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780955    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780958    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780960    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780963    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780966    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780968    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780971    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780974    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.793698 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780976    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780979    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780982    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780986    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780989    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780993    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780995    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.780998    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781000    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781005    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781009    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781012    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781017    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781019    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781022    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781025    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781027    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781030    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781033    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.794357 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781035    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781038    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781040    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781043    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781045    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781048    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781050    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781053    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781055    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781058    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781061    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781063    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781066    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781068    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781071    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781074    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781078    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781080    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781083    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781086    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.794869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781089    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781091    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781094    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781097    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781099    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781103    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781106    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781109    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781111    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781114    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781116    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781119    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781122    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781124    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781126    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781129    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781132    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781134    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781137    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781139    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.795529 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781142    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781144    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781147    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781149    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781152    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781155    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781157    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781160    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781163    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781166    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781168    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781184    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781187    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781190    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781192    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781195    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781198    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781202    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781205    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781207    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.796005 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781210    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781213    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781215    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781218    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781221    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781223    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781227    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.781229    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.785088    4531 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.785105    4531 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.785125    4531 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.785131    4531 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785217    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785224    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785227    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785230    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785233    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785238    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785242    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.796505 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785245    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785248    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785250    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785253    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785256    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785260    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785262    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785265    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785268    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785271    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785274    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785277    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785279    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785282    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785285    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785288    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785291    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785295    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785298    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785300    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.796966 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785303    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785306    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785309    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785312    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785315    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785317    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785320    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785323    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785325    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785328    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785330    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785333    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785336    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785338    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785341    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785343    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785346    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785349    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785388    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785461    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.797513 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785467    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785471    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785476    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785481    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785485    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785490    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785494    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785498    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785503    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785507    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785512    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785516    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785526    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785530    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785535    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785540    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785545    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785549    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785554    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785558    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.804018 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785562    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785567    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785572    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785576    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785586    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785591    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785595    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785600    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785604    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785609    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785613    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785619    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785624    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785629    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785633    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785638    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785642    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785651    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785656    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.804859 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785661    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785665    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785670    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785674    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785679    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785683    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785688    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785692    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785696    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785701    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785705    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785714    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785718    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785722    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785726    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785730    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785735    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785739    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785743    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.805653 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.785747    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.785813    4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786300    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786314    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786319    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786323    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786330    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786334    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786338    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786343    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786347    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786351    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786356    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786360    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786364    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786368    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.806381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786378    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786383    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786387    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786392    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786397    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786401    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786406    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786411    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786415    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786420    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786424    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786428    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786437    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786441    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786446    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786450    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786454    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786458    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786462    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786466    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.806790 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786471    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786476    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786480    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786484    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786489    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786497    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786502    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786506    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786510    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786515    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786520    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786525    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786529    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786533    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786537    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786541    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786545    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786550    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786558    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.807375 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786562    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786567    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786571    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786615    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786633    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786755    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786760    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786764    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786769    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786776    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786780    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786785    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786789    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786793    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786796    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786799    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786802    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786805    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786811    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.807841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786815    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786818    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786820    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786823    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786826    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786829    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786832    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786835    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786838    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786840    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786843    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786846    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786848    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786851    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786854    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786857    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786859    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786862    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786865    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786868    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.808328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786871    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786873    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786876    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786879    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786881    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786884    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786886    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786889    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786892    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786895    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786898    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786901    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786904    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786907    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.786949    4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786989    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.808895 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.786997    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787000    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787003    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787006    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787009    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787012    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787014    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787017    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787020    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787023    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787025    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787029    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787032    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787035    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787038    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787041    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787044    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787046    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787049    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787052    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.809399 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787054    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787057    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787060    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787063    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787065    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787068    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787070    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787073    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787075    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787078    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787080    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787083    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787087    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787089    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787092    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787095    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787098    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787101    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787103    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787106    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.809913 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787108    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787111    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787114    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787116    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787119    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787123    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787126    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787129    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787131    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787134    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787137    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787139    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787142    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787145    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787147    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787150    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787152    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787155    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787157    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787160    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.810654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787163    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787165    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787168    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787192    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787197    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787199    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787202    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787205    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787207    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787210    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787213    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787215    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787218    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787221    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787223    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787226    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787230    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787233    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787238    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.811159 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787242    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787246    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787249    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787253    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787256    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787259    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787261    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787264    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787267    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787269    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787272    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787274    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787277    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787280    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787283    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787286    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787288    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787292    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787295    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787298    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.811725 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787301    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787303    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787306    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787310    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787313    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:26.787316    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.787381    4531 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.788366    4531 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.788442    4531 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.788624    4531 server.go:1014] "Starting client certificate rotation"
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.788765    4531 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.788823    4531 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:25:47 +0000 UTC" deadline="2028-06-23 07:56:14.793222875 +0000 UTC"
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.788853    4531 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14340h24m48.004372385s"
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.789080    4531 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.790285    4531 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.791792    4531 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:31:26.812381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.793808    4531 log.go:25] "Validated CRI v1 image API"
Nov 03 19:31:26.813015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.794572    4531 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:31:26.813015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.796158    4531 fs.go:135] Filesystem UUIDs: map[1e235f90-949f-48e7-aca8-d8d017ee165b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e03333df-c179-4a4d-8e46-8ed103713010:/dev/nvme0n1p3]
Nov 03 19:31:26.813158 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.796194    4531 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/108ce33e60df30154d94da41931e07b9c95caee552fd94e5dff30a69cc8a902b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/108ce33e60df30154d94da41931e07b9c95caee552fd94e5dff30a69cc8a902b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13ea335b9679fd32d02a55062b8596cff1372427eb952cf77fb6b4831b5a8dbe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13ea335b9679fd32d02a55062b8596cff1372427eb952cf77fb6b4831b5a8dbe/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16e2877f87a0befb90ed4eaf6833e145f834da44968f3b1c2bea606ef7a08fc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16e2877f87a0befb90ed4eaf6833e145f834da44968f3b1c2bea606ef7a08fc5/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19c150003ec54bbf8a4ebcdb038974342b4f635d92ee2cffb0fb4bcd864abfa2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19c150003ec54bbf8a4ebcdb038974342b4f635d92ee2cffb0fb4bcd864abfa2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/382413560c9799898e204792bd056dc10a8edc5094dd006f2a3b9a3e78e9ab7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/382413560c9799898e204792bd056dc10a8edc5094dd006f2a3b9a3e78e9ab7d/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40e32d134a03bfbfb2eb354c4a071facdc1ce0bb0ca0c2b301167a47dbc12278/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40e32d134a03bfbfb2eb354c4a071facdc1ce0bb0ca0c2b301167a47dbc12278/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42ca6cbb98a2c823c5bae1cefe5456497ce81993b81d2184264e7c58e99f4783/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42ca6cbb98a2c823c5bae1cefe5456497ce81993b81d2184264e7c58e99f4783/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c63ffbd45c6598a06ba4c7a38bd2f6500230e2a127db1cdad004a6cd8b551e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c63ffbd45c6598a06ba4c7a38bd2f6500230e2a127db1cdad004a6cd8b551e2/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dfe88d7dd01e5ef2eed0e162a001efce1646b7dd90739005770c285a18a6a6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dfe88d7dd01e5ef2eed0e162a001efce1646b7dd90739005770c285a18a6a6f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86baadc1f9343dc7e0c95a6855914e37e2fe2fed21b8393f93cbc95b50c11fed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86baadc1f9343dc7e0c95a6855914e37e2fe2fed21b8393f93cbc95b50c11fed/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96e7e5a9ca903443b08336a03d0fc43666a20f0c2426db9f12505183d30acf0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96e7e5a9ca903443b08336a03d0fc43666a20f0c2426db9f12505183d30acf0a/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3e76e61fa30d5b74144b99cbe768ed6e805d24628f2f34a87e4d2b065e6881e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3e76e61fa30d5b74144b99cbe768ed6e805d24628f2f34a87e4d2b065e6881e/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db03ac7844ab2ae3c86f30aa6077e8853ff741a9d59ffb08f9c919d9add35679/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db03ac7844ab2ae3c86f30aa6077e8853ff741a9d59ffb08f9c919d9add35679/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dff09f0062277125e74fe5e5fb9224577cbceb7a1d19a7a3c8e164d85ab7ddf5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dff09f0062277125e74fe5e5fb9224577cbceb7a1d19a7a3c8e164d85ab7ddf5/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4e8c086238cfe606828d2c4c2b7eed7db3f8ba5d6e960919ea42e279814161d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4e8c086238cfe606828d2c4c2b7eed7db3f8ba5d6e960919ea42e279814161d/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eca8f0b225ae055628e6816ad6c0cfd08ac606a88a32ac697ffbb024772605ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eca8f0b225ae055628e6816ad6c0cfd08ac606a88a32ac697ffbb024772605ca/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a4d21a2-f237-4ffb-88c9-1855e8b4b887/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0a4d21a2-f237-4ffb-88c9-1855e8b4b887/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1697fe5f-c2b8-46e3-b57d-73afd75e6926/volumes/kubernetes.io~projected/kube-api-access-z7srz:{mountpoint:/var/lib/kubelet/pods/1697fe5f-c2b8-46e3-b57d-73afd75e6926/volumes/kubernetes.io~projected/kube-api-access-z7srz major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d4473c0-57b5-44a8-a467-d6a29d958c68/volumes/kubernetes.io~projected/kube-api-access-jzlmm:{mountpoint:/var/lib/kubelet/pods/1d4473c0-57b5-44a8-a467-d6a29d958c68/volumes/kubernetes.io~projected/kube-api-access-jzlmm major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/319a92b5-37b5-40b2-88cd-5550aab72650/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/319a92b5-37b5-40b2-88cd-5550aab72650/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34118633-5be6-49c0-9663-8c66046d847a/volumes/kubernetes.io~projected/kube-api-access-8r9kw:{mountpoint:/var/lib/kubelet/pods/34118633-5be6-49c0-9663-8c66046d847a/volumes/kubernetes.io~projected/kube-api-access-8r9kw major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/448841dc-bb08-4989-8180-abe966bcaab4/volumes/kubernetes.io~projected/kube-api-access-k4bzk:{mountpoint:/var/lib/kubelet/pods/448841dc-bb08-4989-8180-abe966bcaab4/volumes/kubernetes.io~projected/kube-api-access-k4bzk major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44c7af65-aff0-41b1-8f98-2adf11d53954/volumes/kubernetes.io~projected/kube-api-access-nbldx:{mountpoint:/var/lib/kubelet/pods/44c7af65-aff0-41b1-8f98-2adf11d53954/volumes/kubernetes.io~projected/kube-api-access-nbldx major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/volumes/kubernetes.io~projected/kube-api-access-6ppqh:{mountpoint:/var/lib/kubelet/pods/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/volumes/kubernetes.io~projected/kube-api-access-6ppqh major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c63d965-5a4f-481a-8b25-0ac731629615/volumes/kubernetes.io~projected/kube-api-access-ttftp:{mountpoint:/var/lib/kubelet/pods/8c63d965-5a4f-481a-8b25-0ac731629615/volumes/kubernetes.io~projected/kube-api-access-ttftp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f387957-9771-467e-9053-994d10fc652a/volumes/kubernetes.io~projected/kube-api-access-w6xff:{mountpoint:/var/lib/kubelet/pods/8f387957-9771-467e-9053-994d10fc652a/volumes/kubernetes.io~projected/kube-api-access-w6xff major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99d9bea7-e81a-4fd7-8757-1eadfe124622/volumes/kubernetes.io~projected/kube-api-access-2p9qg:{mountpoint:/var/lib/kubelet/pods/99d9bea7-e81a-4fd7-8757-1eadfe124622/volumes/kubernetes.io~projected/kube-api-access-2p9qg major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99d9bea7-e81a-4fd7-8757-1eadfe124622/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/99d9bea7-e81a-4fd7-8757-1eadfe124622/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9abeb54a-e625-4ad1-827e-49f5640d6288/volumes/kubernetes.io~projected/kube-api-access-lx2mr:{mountpoint:/var/lib/kubelet/pods/9abeb54a-e625-4ad1-827e-49f5640d6288/volumes/kubernetes.io~projected/kube-api-access-lx2mr major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab89d762-778b-4e85-b506-72be48727030/volumes/kubernetes.io~projected/kube-api-access-hz5nb:{mountpoint:/var/lib/kubelet/pods/ab89d762-778b-4e85-b506-72be48727030/volumes/kubernetes.io~projected/kube-api-access-hz5nb major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad438ca5-2f83-4e9b-8058-69edd81f888e/volumes/kubernetes.io~projected/kube-api-access-6mfrm:{mountpoint:/var/lib/kubelet/pods/ad438ca5-2f83-4e9b-8058-69edd81f888e/volumes/kubernetes.io~projected/kube-api-access-6mfrm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adabc7a3-99a6-43f0-b1ed-8a0395d310a4/volumes/kubernetes.io~projected/kube-api-access-wvc7b:{mountpoint:/var/lib/kubelet/pods/adabc7a3-99a6-43f0-b1ed-8a0395d310a4/volumes/kubernetes.io~projected/kube-api-access-wvc7b major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2d3316-f9d0-4812-89a3-d34efcc04628/volumes/kubernetes.io~projected/kube-api-access-xng5d:{mountpoint:/var/lib/kubelet/pods/ba2d3316-f9d0-4812-89a3-d34efcc04628/volumes/kubernetes.io~projected/kube-api-access-xng5d major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbbb1119-eb4e-48f1-8d3c-2c9264080717/volumes/kubernetes.io~projected/kube-api-access-nwvm6:{mountpoint:/var/lib/kubelet/pods/bbbb1119-eb4e-48f1-8d3c-2c9264080717/volumes/kubernetes.io~projected/kube-api-access-nwvm6 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbbb1119-eb4e-48f1-8d3c-2c9264080717/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/bbbb1119-eb4e-48f1-8d3c-2c9264080717/volumes/kubernetes.io~secret/signing-key major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c455037a-fc47-4c3d-a917-854f924a5df0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c455037a-fc47-4c3d-a917-854f924a5df0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c455037a-fc47-4c3d-a917-854f924a5df0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c455037a-fc47-4c3d-a917-854f924a5df0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c455037a-fc47-4c3d-a917-854f924a5df0/volumes/kubernetes.io~projected/kube-api-access-frtjc:{mountpoint:/var/lib/kubelet/pods/c455037a-fc47-4c3d-a917-854f924a5df0/volumes/kubernetes.io~projected/kube-api-access-frtjc major:0 minor:53 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/04d20e1ae6404c938b94867267c70d06fa96b9db379b7aa97401607154a0d512/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5b073d2f983e2e7bbfc72d088512ad1f481b9787bc6eb5ee155e32a4456a47c1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/871f8020b30f8c729bfcc586bbba3b7e7398349aedc1510b79f0ec395d40b4ed/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/27f5f5cf5b4d2f64115b9863ae3ddff4038869552dbd57ca2dc7a9dbd71f90da/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/95a24f09da111fcde0bd28012a1fd4191823f8db354e3b302b9b6b3a8bf43227/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8ced58cbb5761de27e95dd121eba7244f8be3ab791e683515a456934dbfcaa5a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-123:{mountpoint:/var/lib/containers/storage/overlay/0c20951e829018fd37795233343a9e66a7d24673150eae0c83266ca1063613cc/merged major:0 minor:123 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/824e78f84d38d445c7b42d5d47096049aba106a3003e13e722d11f68f813566d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8585d5ddfb882a4e1984d6bc9b97ee07b9f5261f1d1b13aaa06543d72d4823c0/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/426f9565cd477de1a9d811606e3fdadfbe2265778471e3db540e05816f39d9d0/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/11ee48230d763716e2ea0d0b117c1c98d8e056053247f110ea51af8e25dcd840/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/88bbf7ad30d5e2f9b3c9476e6e5aaa8467bad78bf71da94de331f69f64d12b59/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a77a32e1377d756dac7e108f112ee4064293248a6df252bdacfddc0f6de37960/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/c9b75643644e22a3e10a4e6f7bd7435026da24c036aaa3cddf2ac6644c27fdeb/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/22b25d403ff8bc847ca26fc45a0f3ba3bab773af21e267a3f3af6ab9aac7ba73/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/eaf4233cf6dd5970b8e5905c3eb95e82a7bb0777d3cd428a6e05d627559b640e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3c0b57066520a373a645c3783453c389882a938eed2c0a39bc274979ba9eb006/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/388b740a16a05b38cbe8546880c06bce2610218a9e84cb6d5c294a7163628099/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2ee5905d0c0bee72474ea9594881b313ffa19fc1cf821041f1ea6f8fdc70fa0f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3a0cd3f56a2b66aee4c6b198e1f247b5c8de6e5842048b910ab5af75414ae48f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7b3896b743704526ef60668549c83a75543d55967e4a15c6acdee540e89212f9/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/0f44d58e05575c8e0d2c04dbc2c92824b8f1e5fef096612b020126a3f4380db8/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/192d89e328bf544127c23c17c06d4f2803e70813804e5be4edce8cb3623dc91f/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/620d176d756b8168002de336cb3e423d631e3f513e2be3150c2cb4a9643cba10/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/11e1d1242d484bd9133ceb5bdcc3f0517d2f6f14ecbfe03803b0838c6dd3bb28/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/16597c79e90141a83c78dc5b1f1d6f01013ca96589fe4aa8f0d3fe7c7bc8c050/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/ed364266da4816a78769fde2deca3f76e3ff2257b8bf3642778972f5b6659d85/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/a4f67f7bea400d4414dc1613a5b181a919368b8342e9532cf362aae01de966fd/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/ff2ee8e5694f70e1628c319be59b1c3b8afa74f1fd1978c4a250c1e9c8ec1533/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/cf96dc8351049956700bf656620143e5cce51a75ba6b2e8c1071766685c19cf5/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/dbfdff54f4d30b88ddbb9b166fe039e3fd6a7e161cd374409b7a986f4e5e238b/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/4824fbb2595fe849770846d96d79d107f8e8f98557f56bd359c097089031759f/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/9246366410d35d0557554804be28401d07316a44b59473bb2c8fcbfae1974759/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/80304d4c855c8612a812420be0532fd9f09b0017d1cf6d047d50dbbb58d00118/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/ed7f5df11b75a12d260d311d6d7c60dea6ceb9a1128ad0f9e781aaa8854f596d/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/083a558ba151c324406c38ddfe317b5ee670cd647420594aea2c2a2ea98b2a26/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8d91662b5759ee86cf190275f80e415eb9ded42ad04738d01b7167669b37fba6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/2c0f3712b391619f8f60f599b60f7e96ac74d209902c5b5e00730d1f00d35585/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/98a59782bd1390fe39200183fa7469400b030d206a8f5ff7242971faf592cd4f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/df0f4b90bba38e58cd836b39c19e92fff66535871ca63350b66d872aafbe77bf/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/c40ddcbb7c42145cd695592feeb15151c8e1f9cb0fcec5406d1c7ef4a7310a8b/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/7e7b0b6804c8d667b7e96dfe9a70417dc51154c75c2563e2c6e3c52d693b6bc3/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/24472d99086d146a575d6f9c9989a6d51aeeb3d551b9f7e871f007286536dcda/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 19:31:26.813152 ip-10-0-131-234 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.803127    4531 manager.go:217] Machine: {Timestamp:2026-11-03 19:31:26.802187802 +0000 UTC m=+0.072311093 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100154 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a09d085c1a356e076734ee67ac289 SystemUUID:ec2a09d0-85c1-a356-e076-734ee67ac289 BootID:865fa0b7-6663-4b22-b9fe-52cc48348515 Filesystems:[{Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9abeb54a-e625-4ad1-827e-49f5640d6288/volumes/kubernetes.io~projected/kube-api-access-lx2mr DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c455037a-fc47-4c3d-a917-854f924a5df0/volumes/kubernetes.io~projected/kube-api-access-frtjc DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13ea335b9679fd32d02a55062b8596cff1372427eb952cf77fb6b4831b5a8dbe/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a4d21a2-f237-4ffb-88c9-1855e8b4b887/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/448841dc-bb08-4989-8180-abe966bcaab4/volumes/kubernetes.io~projected/kube-api-access-k4bzk DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4e8c086238cfe606828d2c4c2b7eed7db3f8ba5d6e960919ea42e279814161d/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/382413560c9799898e204792bd056dc10a8edc5094dd006f2a3b9a3e78e9ab7d/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db03ac7844ab2ae3c86f30aa6077e8853ff741a9d59ffb08f9c919d9add35679/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bbbb1119-eb4e-48f1-8d3c-2c9264080717/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab89d762-778b-4e85-b506-72be48727030/volumes/kubernetes.io~projected/kube-api-access-hz5nb DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42ca6cbb98a2c823c5bae1cefe5456497ce81993b81d2184264e7c58e99f4783/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bbbb1119-eb4e-48f1-8d3c-2c9264080717/volumes/kubernetes.io~projected/kube-api-access-nwvm6 DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1697fe5f-c2b8-46e3-b57d-73afd75e6926/volumes/kubernetes.io~projected/kube-api-access-z7srz DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eca8f0b225ae055628e6816ad6c0cfd08ac606a88a32ac697ffbb024772605ca/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16e2877f87a0befb90ed4eaf6833e145f834da44968f3b1c2bea606ef7a08fc5/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34118633-5be6-49c0-9663-8c66046d847a/volumes/kubernetes.io~projected/kube-api-access-8r9kw DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/c455037a-fc47-4c3d-a917-854f924a5df0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f387957-9771-467e-9053-994d10fc652a/volumes/kubernetes.io~projected/kube-api-access-w6xff DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/volumes/kubernetes.io~projected/kube-api-access-6ppqh DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99d9bea7-e81a-4fd7-8757-1eadfe124622/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40e32d134a03bfbfb2eb354c4a071facdc1ce0bb0ca0c2b301167a47dbc12278/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/44c7af65-aff0-41b1-8f98-2adf11d53954/volumes/kubernetes.io~projected/kube-api-access-nbldx DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2d3316-f9d0-4812-89a3-d34efcc04628/volumes/kubernetes.io~projected/kube-api-access-xng5d DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ad438ca5-2f83-4e9b-8058-69edd81f888e/volumes/kubernetes.io~projected/kube-api-access-6mfrm DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dff09f0062277125e74fe5e5fb9224577cbceb7a1d19a7a3c8e164d85ab7ddf5/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c455037a-fc47-4c3d-a917-854f924a5df0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c63ffbd45c6598a06ba4c7a38bd2f6500230e2a127db1cdad004a6cd8b551e2/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99d9bea7-e81a-4fd7-8757-1eadfe124622/volumes/kubernetes.io~projected/kube-api-access-2p9qg DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/319a92b5-37b5-40b2-88cd-5550aab72650/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96e7e5a9ca903443b08336a03d0fc43666a20f0c2426db9f12505183d30acf0a/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adabc7a3-99a6-43f0-b1ed-8a0395d310a4/volumes/kubernetes.io~projected/kube-api-access-wvc7b DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dfe88d7dd01e5ef2eed0e162a001efce1646b7dd90739005770c285a18a6a6f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c63d965-5a4f-481a-8b25-0ac731629615/volumes/kubernetes.io~projected/kube-api-access-ttftp DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3e76e61fa30d5b74144b99cbe768ed6e805d24628f2f34a87e4d2b065e6881e/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86baadc1f9343dc7e0c95a6855914e37e2fe2fed21b8393f93cbc95b50c11fed/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19c150003ec54bbf8a4ebcdb038974342b4f635d92ee2cffb0fb4bcd864abfa2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d4473c0-57b5-44a8-a467-d6a29d958c68/volumes/kubernetes.io~projected/kube-api-access-jzlmm DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/108ce33e60df30154d94da41931e07b9c95caee552fd94e5dff30a69cc8a902b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-123 DeviceMajor:0 DeviceMinor:123 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:16e2877f87a0bef MacAddress:ca:2f:e5:e5:bb:95 Speed:10000 Mtu:8901} {Name:40e32d134a03bfb MacAddress:02:91:eb:70:d2:bb Speed:10000 Mtu:8901} {Name:42ca6cbb98a2c82 MacAddress:a2:a7:0b:54:16:03 Speed:10000 Mtu:8901} {Name:96e7e5a9ca90344 MacAddress:b2:e4:95:e5:63:e9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f4:30:cf:6d Speed:0 Mtu:9001} {Name:br-int MacAddress:de:71:e2:20:d3:70 Speed:0 Mtu:8901} {Name:db03ac7844ab2ae MacAddress:d6:6e:62:1b:68:35 Speed:10000 Mtu:8901} {Name:dff09f006227712 MacAddress:aa:a9:2e:ea:fb:3c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f4:30:cf:6d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:3e:53:de:f6:2e:dd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fe:74:84:ae:89:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.803594    4531 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.803665    4531 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.803998    4531 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804024    4531 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-234.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804234    4531 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804247    4531 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804262    4531 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804292    4531 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804439    4531 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804658    4531 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804803    4531 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804849    4531 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804867    4531 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804881    4531 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804893    4531 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.804906    4531 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.805729    4531 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.805743    4531 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.806326    4531 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808344    4531 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808354    4531 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808448    4531 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808644    4531 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808671    4531 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808761    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808773    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808779    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808785    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808791    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808796    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808802    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808808    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808815    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808827    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808840    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808849    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808872    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.808878    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.812409    4531 server.go:1263] "Started kubelet"
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.812453    4531 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.812588    4531 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:31:26.813600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.812680    4531 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:31:26.815787 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.814132    4531 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:31:26.815787 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.815336    4531 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:31:26.815787 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.815393    4531 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:31:26.815787 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.815377    4531 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:31:26.815787 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.815414    4531 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:31:26.815787 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:26.815665    4531 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-234.us-east-2.compute.internal\" not found"
Nov 03 19:31:26.816044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.815405    4531 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:31:26.816044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.815939    4531 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:25:48 +0000 UTC" deadline="2028-08-07 17:36:14.455649904 +0000 UTC"
Nov 03 19:31:26.816044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.815981    4531 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15430h4m47.639673121s"
Nov 03 19:31:26.817160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.816699    4531 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:31:26.824006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.818418    4531 factory.go:153] Registering CRI-O factory
Nov 03 19:31:26.824006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.818439    4531 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:31:26.824006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.818603    4531 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:31:26.824006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.818615    4531 factory.go:55] Registering systemd factory
Nov 03 19:31:26.824006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.818625    4531 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:31:26.824006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.818643    4531 factory.go:103] Registering Raw factory
Nov 03 19:31:26.824006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.818665    4531 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:31:26.824006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.819351    4531 manager.go:319] Starting recovery of all containers
Nov 03 19:31:26.824006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.821641    4531 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:31:26.824006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.823063    4531 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:31:26.824684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.824033    4531 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:31:26.825244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825191    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d4473c0-57b5-44a8-a467-d6a29d958c68" volumeName="kubernetes.io/empty-dir/1d4473c0-57b5-44a8-a467-d6a29d958c68-tmp-dir" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825247    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34118633-5be6-49c0-9663-8c66046d847a" volumeName="kubernetes.io/configmap/34118633-5be6-49c0-9663-8c66046d847a-iptables-alerter-script" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825261    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c455037a-fc47-4c3d-a917-854f924a5df0" volumeName="kubernetes.io/projected/c455037a-fc47-4c3d-a917-854f924a5df0-kube-api-access-frtjc" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825274    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d4473c0-57b5-44a8-a467-d6a29d958c68" volumeName="kubernetes.io/projected/1d4473c0-57b5-44a8-a467-d6a29d958c68-kube-api-access-jzlmm" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825286    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34118633-5be6-49c0-9663-8c66046d847a" volumeName="kubernetes.io/projected/34118633-5be6-49c0-9663-8c66046d847a-kube-api-access-8r9kw" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825301    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="448841dc-bb08-4989-8180-abe966bcaab4" volumeName="kubernetes.io/projected/448841dc-bb08-4989-8180-abe966bcaab4-kube-api-access-k4bzk" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825312    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44c7af65-aff0-41b1-8f98-2adf11d53954" volumeName="kubernetes.io/projected/44c7af65-aff0-41b1-8f98-2adf11d53954-kube-api-access-nbldx" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825325    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1" volumeName="kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-env-overrides" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825339    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad438ca5-2f83-4e9b-8058-69edd81f888e" volumeName="kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825351    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1697fe5f-c2b8-46e3-b57d-73afd75e6926" volumeName="kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825363    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f387957-9771-467e-9053-994d10fc652a" volumeName="kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825376    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad438ca5-2f83-4e9b-8058-69edd81f888e" volumeName="kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825387    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adabc7a3-99a6-43f0-b1ed-8a0395d310a4" volumeName="kubernetes.io/configmap/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-serviceca" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825399    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adabc7a3-99a6-43f0-b1ed-8a0395d310a4" volumeName="kubernetes.io/projected/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-kube-api-access-wvc7b" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825430    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2d3316-f9d0-4812-89a3-d34efcc04628" volumeName="kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-service-ca-bundle" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825442    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a4d21a2-f237-4ffb-88c9-1855e8b4b887" volumeName="kubernetes.io/configmap/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-konnectivity-ca" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825471    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a4d21a2-f237-4ffb-88c9-1855e8b4b887" volumeName="kubernetes.io/secret/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-agent-certs" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825494    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1697fe5f-c2b8-46e3-b57d-73afd75e6926" volumeName="kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-daemon-config" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825508    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1" volumeName="kubernetes.io/secret/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825519    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c63d965-5a4f-481a-8b25-0ac731629615" volumeName="kubernetes.io/empty-dir/8c63d965-5a4f-481a-8b25-0ac731629615-tmp-dir" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825531    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c63d965-5a4f-481a-8b25-0ac731629615" volumeName="kubernetes.io/projected/8c63d965-5a4f-481a-8b25-0ac731629615-kube-api-access-ttftp" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825542    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99d9bea7-e81a-4fd7-8757-1eadfe124622" volumeName="kubernetes.io/secret/99d9bea7-e81a-4fd7-8757-1eadfe124622-cert" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825555    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad438ca5-2f83-4e9b-8058-69edd81f888e" volumeName="kubernetes.io/projected/ad438ca5-2f83-4e9b-8058-69edd81f888e-kube-api-access-6mfrm" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825569    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="319a92b5-37b5-40b2-88cd-5550aab72650" volumeName="kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825582    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1" volumeName="kubernetes.io/projected/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-kube-api-access-6ppqh" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825594    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9abeb54a-e625-4ad1-827e-49f5640d6288" volumeName="kubernetes.io/projected/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-api-access-lx2mr" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825607    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2d3316-f9d0-4812-89a3-d34efcc04628" volumeName="kubernetes.io/empty-dir/ba2d3316-f9d0-4812-89a3-d34efcc04628-archives-path" seLinuxMountContext=""
Nov 03 19:31:26.825607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825619    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2d3316-f9d0-4812-89a3-d34efcc04628" volumeName="kubernetes.io/projected/ba2d3316-f9d0-4812-89a3-d34efcc04628-kube-api-access-xng5d" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825640    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c455037a-fc47-4c3d-a917-854f924a5df0" volumeName="kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-tmp" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825657    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1" volumeName="kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-config" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825670    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1697fe5f-c2b8-46e3-b57d-73afd75e6926" volumeName="kubernetes.io/projected/1697fe5f-c2b8-46e3-b57d-73afd75e6926-kube-api-access-z7srz" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825681    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99d9bea7-e81a-4fd7-8757-1eadfe124622" volumeName="kubernetes.io/projected/99d9bea7-e81a-4fd7-8757-1eadfe124622-kube-api-access-2p9qg" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825749    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2d3316-f9d0-4812-89a3-d34efcc04628" volumeName="kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825762    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbbb1119-eb4e-48f1-8d3c-2c9264080717" volumeName="kubernetes.io/secret/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-key" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825775    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d4473c0-57b5-44a8-a467-d6a29d958c68" volumeName="kubernetes.io/configmap/1d4473c0-57b5-44a8-a467-d6a29d958c68-config-volume" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825787    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab89d762-778b-4e85-b506-72be48727030" volumeName="kubernetes.io/projected/ab89d762-778b-4e85-b506-72be48727030-kube-api-access-hz5nb" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825797    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad438ca5-2f83-4e9b-8058-69edd81f888e" volumeName="kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825808    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbbb1119-eb4e-48f1-8d3c-2c9264080717" volumeName="kubernetes.io/projected/bbbb1119-eb4e-48f1-8d3c-2c9264080717-kube-api-access-nwvm6" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825820    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1" volumeName="kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825832    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9abeb54a-e625-4ad1-827e-49f5640d6288" volumeName="kubernetes.io/empty-dir/9abeb54a-e625-4ad1-827e-49f5640d6288-data-volume" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825843    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbbb1119-eb4e-48f1-8d3c-2c9264080717" volumeName="kubernetes.io/configmap/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-cabundle" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825855    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c455037a-fc47-4c3d-a917-854f924a5df0" volumeName="kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-etc-tuned" seLinuxMountContext=""
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825867    4531 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:31:26.826717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.825876    4531 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:31:26.828081 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.827731    4531 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 19:31:26.843616 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.843592    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:31:26.844029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.844006    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:31:26.848976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.848945    4531 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:31:26.850388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.850365    4531 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:31:26.850498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.850396    4531 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:31:26.850498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.850420    4531 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:31:26.850498 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:26.850453    4531 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 19:31:26.852864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.852843    4531 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:31:26.855278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.855250    4531 generic.go:356] "Generic (PLEG): container finished" podID="50b103dd723deeb9c86a7ed93f39fb9c" containerID="01201bacd0aeab71e6c83a8e706ba7fd408437df986516d3212535780305ede3" exitCode=0
Nov 03 19:31:26.877250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.877224    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:31:26.877566 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.877541    4531 generic.go:356] "Generic (PLEG): container finished" podID="4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1" containerID="8401c705e45efa0503aacd9e2c77babf4de46b83f7a64a6917da1474d50b77d1" exitCode=1
Nov 03 19:31:26.890373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.890344    4531 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="e03be54ddbfa0bd66c0a9fd205ef44d62e1cd21fdb1508d48323ed4f50372b99" exitCode=0
Nov 03 19:31:26.890373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.890371    4531 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="75deb1e18366bed76ad76d48dde808b000899d3cc037c3e9527038bc8837714d" exitCode=0
Nov 03 19:31:26.890373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.890379    4531 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="ff2d5f56cf4d107ea2800409b3f6be160cd4640e7b67b2daf71cb1fe37a40478" exitCode=0
Nov 03 19:31:26.890561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.890386    4531 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="3ef77b8f45a694988c6e79088dd00cb5013eb47dd099d95a1aedf3a956d34837" exitCode=0
Nov 03 19:31:26.890561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.890395    4531 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="dffcf25959aea887b99a007709d78a0998c119cdcc402c25fece6ac7dba10310" exitCode=0
Nov 03 19:31:26.890561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.890401    4531 generic.go:356] "Generic (PLEG): container finished" podID="ad438ca5-2f83-4e9b-8058-69edd81f888e" containerID="fda3c7f10fe670f644f02dff5bc95dc8a6a19b8d9fee89abd5f8098dd4d5c97c" exitCode=0
Nov 03 19:31:26.895831 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.895808    4531 manager.go:324] Recovery completed
Nov 03 19:31:26.911135 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.911066    4531 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:31:26.911135 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.911082    4531 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:31:26.911135 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.911102    4531 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:31:26.911373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.911269    4531 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 19:31:26.911764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.911284    4531 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 19:31:26.911850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.911766    4531 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 19:31:26.911850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.911775    4531 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 19:31:26.911850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.911783    4531 policy_none.go:50] "Start"
Nov 03 19:31:26.911850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.911796    4531 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:31:26.911850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.911811    4531 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:31:26.912088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.912001    4531 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 19:31:26.912088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.912013    4531 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 19:31:26.912088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.912021    4531 policy_none.go:44] "Start"
Nov 03 19:31:26.916147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.916120    4531 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:31:26.920738 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.920721    4531 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:31:26.920824 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:26.920751    4531 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:31:26.920824 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.920763    4531 server.go:87] "Starting device plugin registration server"
Nov 03 19:31:26.921090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.921058    4531 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:31:26.921090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.921072    4531 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:31:26.921220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.921168    4531 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:31:26.921323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.921286    4531 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:31:26.921362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.921331    4531 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:31:26.921362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.921338    4531 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:31:26.921462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.921386    4531 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:31:26.921321442Z","UUID":"b3259ffc-164e-41c1-aea6-dc80f0f3d081","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:31:26.922598 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:26.922578    4531 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:31:26.922772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.922744    4531 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:31:26.922772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.922770    4531 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:31:26.951124 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.951073    4531 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"]
Nov 03 19:31:26.951601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.951571    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal" event={"ID":"50b103dd723deeb9c86a7ed93f39fb9c","Type":"ContainerStarted","Data":"eafedaf238ee2922474cafb4a0a240fb18935083b5eae1bbee41c79122ea37e3"}
Nov 03 19:31:26.951638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.951608    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal" event={"ID":"50b103dd723deeb9c86a7ed93f39fb9c","Type":"ContainerDied","Data":"01201bacd0aeab71e6c83a8e706ba7fd408437df986516d3212535780305ede3"}
Nov 03 19:31:26.951638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.951624    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal" event={"ID":"50b103dd723deeb9c86a7ed93f39fb9c","Type":"ContainerStarted","Data":"e4e8c086238cfe606828d2c4c2b7eed7db3f8ba5d6e960919ea42e279814161d"}
Nov 03 19:31:26.951711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.951645    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal" event={"ID":"d9cc75c72d3d8481cfe8602c9dd164cf","Type":"ContainerStarted","Data":"ff892b49bec1886c11603ca64f378e4114e2daaa32e15f174fc41c35c00757ed"}
Nov 03 19:31:26.951711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.951656    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal" event={"ID":"d9cc75c72d3d8481cfe8602c9dd164cf","Type":"ContainerStarted","Data":"4c63ffbd45c6598a06ba4c7a38bd2f6500230e2a127db1cdad004a6cd8b551e2"}
Nov 03 19:31:26.951711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.951681    4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:26.951906 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.951889    4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:26.962905 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.962887    4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:31:26.962973 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:26.962933    4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:26.963341 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:26.963327    4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:31:26.963381 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:26.963362    4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.021667 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.021635    4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:31:27.023003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.022983    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:31:27.023092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.023016    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:31:27.023092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.023032    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-234.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:31:27.023092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.023071    4531 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.028281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.028260    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.028344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.028288    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.028344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.028307    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9cc75c72d3d8481cfe8602c9dd164cf-config\") pod \"kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"d9cc75c72d3d8481cfe8602c9dd164cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.034317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.034299    4531 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.034398 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.034365    4531 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.128495 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.128456    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.128495 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.128497    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9cc75c72d3d8481cfe8602c9dd164cf-config\") pod \"kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"d9cc75c72d3d8481cfe8602c9dd164cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.128711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.128523    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.128711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.128563    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.128711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.128563    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50b103dd723deeb9c86a7ed93f39fb9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"50b103dd723deeb9c86a7ed93f39fb9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.128711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.128577    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9cc75c72d3d8481cfe8602c9dd164cf-config\") pod \"kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal\" (UID: \"d9cc75c72d3d8481cfe8602c9dd164cf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.805503 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.805466    4531 apiserver.go:52] "Watching apiserver"
Nov 03 19:31:27.810723 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.810695    4531 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:31:27.811263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.811240    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-vj6pd","openshift-dns/node-resolver-q22m7","openshift-multus/multus-additional-cni-plugins-w2kqz","kube-system/konnectivity-agent-4jl9f","kube-system/kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-m4rw5","openshift-ovn-kubernetes/ovnkube-node-dqz7h","openshift-insights/insights-runtime-extractor-rjzpc","openshift-ingress-canary/ingress-canary-8pptd","openshift-insights/periodic-gathering-9xxl7-qt48t","openshift-multus/network-metrics-daemon-mgl8h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal","openshift-service-ca/service-ca-5fffc97bdf-pdfb6","openshift-cluster-node-tuning-operator/tuned-mfjxm","openshift-dns/dns-default-tzpld","openshift-image-registry/node-ca-gcvlj","openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4","openshift-network-operator/iptables-alerter-qmfcq","openshift-multus/multus-vbrfn"]
Nov 03 19:31:27.811686 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.811666    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:27.815225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.815202    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:31:27.815636 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.815612    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.815759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.815694    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.815759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.815707    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:31:27.816452 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.816415    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:31:27.816577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.816429    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.817007 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.816947    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.817257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.817238    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.817782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.817747    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:31:27.818115 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.818093    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:31:27.818247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.818223    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9h8df\""
Nov 03 19:31:27.818479 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.818460    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.818588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.818574    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:31:27.818651 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.818626    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wllhj\""
Nov 03 19:31:27.818811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.818795    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:31:27.819035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.818824    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-glk7q\""
Nov 03 19:31:27.819035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.818983    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tdq2h\""
Nov 03 19:31:27.819035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.819023    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:31:27.819035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.819033    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:31:27.819254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.819066    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:31:27.819307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.819276    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:31:27.819649 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.819631    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.819779 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.819763    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.819967 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.819950    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:31:27.820061 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.820047    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.820727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.820710    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:31:27.820920 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.820901    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k6x7r\""
Nov 03 19:31:27.821416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.821394    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-77jlf\""
Nov 03 19:31:27.821528 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.821493    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.822647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.822626    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.822719 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.822709    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:31:27.822767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.822709    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kqxn2\""
Nov 03 19:31:27.822767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.822742    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vgldx\""
Nov 03 19:31:27.822874 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.822848    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.823098 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.823083    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:27.823220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.823201    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pq7nf\""
Nov 03 19:31:27.823326 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.823313    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bh6bb\""
Nov 03 19:31:27.823421 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.823405    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.824627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.824603    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.824973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.824953    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khl78\""
Nov 03 19:31:27.825210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.825191    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:31:27.825376 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.825361    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:31:27.825597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.825570    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:27.825957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.825939    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:27.826133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826113    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.826223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826152    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:31:27.826271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826224    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-brhwf\""
Nov 03 19:31:27.826381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826351    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6p8wh\""
Nov 03 19:31:27.826478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826437    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.826530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826481    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.826530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826515    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.826604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826551    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.826604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826561    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:31:27.826668 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826622    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-skb8s\""
Nov 03 19:31:27.826775 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826762    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2lfv2\""
Nov 03 19:31:27.826883 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.826873    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.827265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.827245    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 19:31:27.827347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.827311    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.827434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.827421    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.827489 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.827480    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gbtlj\""
Nov 03 19:31:27.827868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.827856    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 19:31:27.828246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.828231    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 19:31:27.828306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.828287    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.828354 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.828345    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.828606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.828592    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wrft7\""
Nov 03 19:31:27.829932 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.829913    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:31:27.829932 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.829916    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:31:27.830067 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.830010    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nblsj\""
Nov 03 19:31:27.830485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.830468    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 19:31:27.833973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.833958    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-var-lib-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.834032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.833981    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-etc-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.834032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834009    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cni-binary-copy\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.834109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834045    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-etc-kubernetes\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.834182 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834156    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-agent-certs\") pod \"konnectivity-agent-4jl9f\" (UID: \"0a4d21a2-f237-4ffb-88c9-1855e8b4b887\") " pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:31:27.834215 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834196    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c63d965-5a4f-481a-8b25-0ac731629615-tmp-dir\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:31:27.834244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834213    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.834244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834229    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-k8s-cni-cncf-io\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.834339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834245    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-kubelet\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.834339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-hostroot\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.834339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834320    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c63d965-5a4f-481a-8b25-0ac731629615-tmp-dir\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:31:27.834477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834322    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34118633-5be6-49c0-9663-8c66046d847a-iptables-alerter-script\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:31:27.834477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834449    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cni-binary-copy\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.834477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834462    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-systemd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.834669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834489    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-config\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.834669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834529    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.834669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834546    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-device-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.834669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834570    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-system-cni-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.834669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834612    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-agent-certs\") pod \"konnectivity-agent-4jl9f\" (UID: \"0a4d21a2-f237-4ffb-88c9-1855e8b4b887\") " pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:31:27.834868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834679    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.834868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834695    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34118633-5be6-49c0-9663-8c66046d847a-iptables-alerter-script\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:31:27.834868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834715    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.834868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834741    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-config\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.834868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834744    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttftp\" (UniqueName: \"kubernetes.io/projected/8c63d965-5a4f-481a-8b25-0ac731629615-kube-api-access-ttftp\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:31:27.834868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834787    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mfrm\" (UniqueName: \"kubernetes.io/projected/ad438ca5-2f83-4e9b-8058-69edd81f888e-kube-api-access-6mfrm\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.834868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834831    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-netns\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.835133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.834950    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-bin\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.835133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835008    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.835133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835027    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cnibin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.835133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835046    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvc7b\" (UniqueName: \"kubernetes.io/projected/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-kube-api-access-wvc7b\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:31:27.835133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835061    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-env-overrides\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.835353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835168    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-script-lib\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.835353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835193    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-env-overrides\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.835353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835218    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34118633-5be6-49c0-9663-8c66046d847a-host-slash\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:31:27.835353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835244    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-bin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.835353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835308    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-slash\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.835353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835341    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-kubelet\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.835552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835366    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-host\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:31:27.835552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835392    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovnkube-script-lib\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.835552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835404    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-konnectivity-ca\") pod \"konnectivity-agent-4jl9f\" (UID: \"0a4d21a2-f237-4ffb-88c9-1855e8b4b887\") " pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:31:27.835552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835450    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-system-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.835552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835535    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-socket-dir-parent\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.835781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835552    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a4d21a2-f237-4ffb-88c9-1855e8b4b887-konnectivity-ca\") pod \"konnectivity-agent-4jl9f\" (UID: \"0a4d21a2-f237-4ffb-88c9-1855e8b4b887\") " pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:31:27.835781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835561    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-conf-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.835781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835587    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-daemon-config\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.835781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835627    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-sys-fs\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.835781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835697    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbldx\" (UniqueName: \"kubernetes.io/projected/44c7af65-aff0-41b1-8f98-2adf11d53954-kube-api-access-nbldx\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.835781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835724    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-systemd-units\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.835781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835758    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.835781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835774    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-log-socket\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.835781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835771    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-daemon-config\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835796    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovn-node-metrics-cert\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835818    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-multus-certs\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835840    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xff\" (UniqueName: \"kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff\") pod \"network-check-target-m4rw5\" (UID: \"8f387957-9771-467e-9053-994d10fc652a\") " pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835860    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r9kw\" (UniqueName: \"kubernetes.io/projected/34118633-5be6-49c0-9663-8c66046d847a-kube-api-access-8r9kw\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835907    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-netns\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835937    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835961    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-serviceca\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.835952    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836086    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836111    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c63d965-5a4f-481a-8b25-0ac731629615-hosts-file\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836134    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-node-log\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.836205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836195    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-multus\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836228    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7srz\" (UniqueName: \"kubernetes.io/projected/1697fe5f-c2b8-46e3-b57d-73afd75e6926-kube-api-access-z7srz\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836253    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-ovn\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836259    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836290    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-socket-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836310    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-ovn-node-metrics-cert\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836330    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-registration-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836340    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-serviceca\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836371    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836388    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836404    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-netd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836437    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ppqh\" (UniqueName: \"kubernetes.io/projected/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-kube-api-access-6ppqh\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836525    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bzk\" (UniqueName: \"kubernetes.io/projected/448841dc-bb08-4989-8180-abe966bcaab4-kube-api-access-k4bzk\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836542    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-cnibin\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.836623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836557    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.836975 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.836701    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad438ca5-2f83-4e9b-8058-69edd81f888e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.845063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.845036    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bzk\" (UniqueName: \"kubernetes.io/projected/448841dc-bb08-4989-8180-abe966bcaab4-kube-api-access-k4bzk\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:27.845388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.845365    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6xff\" (UniqueName: \"kubernetes.io/projected/8f387957-9771-467e-9053-994d10fc652a-kube-api-access-w6xff\") pod \"network-check-target-m4rw5\" (UID: \"8f387957-9771-467e-9053-994d10fc652a\") " pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:27.845462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.845445    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r9kw\" (UniqueName: \"kubernetes.io/projected/34118633-5be6-49c0-9663-8c66046d847a-kube-api-access-8r9kw\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:31:27.845525 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.845509    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7srz\" (UniqueName: \"kubernetes.io/projected/1697fe5f-c2b8-46e3-b57d-73afd75e6926-kube-api-access-z7srz\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.845942 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.845926    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvc7b\" (UniqueName: \"kubernetes.io/projected/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-kube-api-access-wvc7b\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:31:27.846061 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.846040    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mfrm\" (UniqueName: \"kubernetes.io/projected/ad438ca5-2f83-4e9b-8058-69edd81f888e-kube-api-access-6mfrm\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.846411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.846395    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ppqh\" (UniqueName: \"kubernetes.io/projected/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-kube-api-access-6ppqh\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.847166 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.847150    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbldx\" (UniqueName: \"kubernetes.io/projected/44c7af65-aff0-41b1-8f98-2adf11d53954-kube-api-access-nbldx\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.849829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.849813    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttftp\" (UniqueName: \"kubernetes.io/projected/8c63d965-5a4f-481a-8b25-0ac731629615-kube-api-access-ttftp\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:31:27.899792 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.899760    4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.908310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.908290    4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:31:27.908436 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:27.908329    4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-234.us-east-2.compute.internal"
Nov 03 19:31:27.916606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.916582    4531 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:31:27.936759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936726    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-systemd\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.936759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936763    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-cnibin\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.936908 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936784    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-var-lib-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.936908 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936833    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-etc-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.936908 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936856    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-etc-kubernetes\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.936908 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936872    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.936908 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936888    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-k8s-cni-cncf-io\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936910    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-kubelet\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936924    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-hostroot\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936942    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-service-ca-bundle\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936947    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936965    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-systemd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936986    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-etc-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936997    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-kubelet\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937000    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-systemd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.936949    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-etc-kubernetes\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937005    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-var-lib-kubelet\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937019    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-hostroot\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937039    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p9qg\" (UniqueName: \"kubernetes.io/projected/99d9bea7-e81a-4fd7-8757-1eadfe124622-kube-api-access-2p9qg\") pod \"ingress-canary-8pptd\" (UID: \"99d9bea7-e81a-4fd7-8757-1eadfe124622\") " pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937069    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.937089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937090    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-var-lib-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937113    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-k8s-cni-cncf-io\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937125    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-device-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937142    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-cnibin\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937149    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-system-cni-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937226    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937227    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-system-cni-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937228    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-device-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937223    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937279    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937297    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad438ca5-2f83-4e9b-8058-69edd81f888e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2kqz\" (UID: \"ad438ca5-2f83-4e9b-8058-69edd81f888e\") " pod="openshift-multus/multus-additional-cni-plugins-w2kqz"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937301    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-service-ca-bundle\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937308    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937323    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937334    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-run\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937356    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-sys\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937417    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-netns\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937449    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-run-netns\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937478    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-bin\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937505    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937529    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cnibin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937553    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-bin\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937554    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-trusted-ca-bundle\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937572    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937590    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-conf\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937601    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-cnibin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937637    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34118633-5be6-49c0-9663-8c66046d847a-host-slash\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:31:27.937659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937658    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99d9bea7-e81a-4fd7-8757-1eadfe124622-cert\") pod \"ingress-canary-8pptd\" (UID: \"99d9bea7-e81a-4fd7-8757-1eadfe124622\") " pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937688    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937716    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-bin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937735    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/319a92b5-37b5-40b2-88cd-5550aab72650-original-pull-secret\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937740    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9abeb54a-e625-4ad1-827e-49f5640d6288-crio-socket\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937716    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34118633-5be6-49c0-9663-8c66046d847a-host-slash\") pod \"iptables-alerter-qmfcq\" (UID: \"34118633-5be6-49c0-9663-8c66046d847a\") " pod="openshift-network-operator/iptables-alerter-qmfcq"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937795    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-modprobe-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937821    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-lib-modules\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937825    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-bin\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937844    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-trusted-ca-bundle\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937846    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-host\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937901    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-tmp\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937931    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtjc\" (UniqueName: \"kubernetes.io/projected/c455037a-fc47-4c3d-a917-854f924a5df0-kube-api-access-frtjc\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937937    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99d9bea7-e81a-4fd7-8757-1eadfe124622-cert\") pod \"ingress-canary-8pptd\" (UID: \"99d9bea7-e81a-4fd7-8757-1eadfe124622\") " pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937967    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-slash\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.937989    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-tmp\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938030    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5nb\" (UniqueName: \"kubernetes.io/projected/ab89d762-778b-4e85-b506-72be48727030-kube-api-access-hz5nb\") pod \"migrator-66df5d496b-fk9b4\" (UID: \"ab89d762-778b-4e85-b506-72be48727030\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938059    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-slash\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938093    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9abeb54a-e625-4ad1-827e-49f5640d6288-data-volume\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938113    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-kubelet\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938128    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-host\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938187    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba2d3316-f9d0-4812-89a3-d34efcc04628-archives-path\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938220    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938236    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adabc7a3-99a6-43f0-b1ed-8a0395d310a4-host\") pod \"node-ca-gcvlj\" (UID: \"adabc7a3-99a6-43f0-b1ed-8a0395d310a4\") " pod="openshift-image-registry/node-ca-gcvlj"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938222    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-kubelet\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938249    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-system-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938275    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-socket-dir-parent\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938278    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9abeb54a-e625-4ad1-827e-49f5640d6288-data-volume\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938297    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba2d3316-f9d0-4812-89a3-d34efcc04628-archives-path\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938317    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-conf-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938347    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-system-cni-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938353    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-conf-dir\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938362    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-multus-socket-dir-parent\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938386    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwvm6\" (UniqueName: \"kubernetes.io/projected/bbbb1119-eb4e-48f1-8d3c-2c9264080717-kube-api-access-nwvm6\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938408    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-sys-fs\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938451    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-sys-fs\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938517    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-etc-tuned\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938544    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-systemd-units\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938573    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-log-socket\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938601    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c455037a-fc47-4c3d-a917-854f924a5df0-etc-tuned\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938615    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-log-socket\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938637    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-systemd-units\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938666    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-multus-certs\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938704    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng5d\" (UniqueName: \"kubernetes.io/projected/ba2d3316-f9d0-4812-89a3-d34efcc04628-kube-api-access-xng5d\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938708    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-multus-certs\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938819    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-netns\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938838    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-kubelet-config\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938856    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938875    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4473c0-57b5-44a8-a467-d6a29d958c68-config-volume\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938878    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-run-netns\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.938894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938896    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4473c0-57b5-44a8-a467-d6a29d958c68-tmp-dir\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.938994    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4473c0-57b5-44a8-a467-d6a29d958c68-tmp-dir\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939035    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlmm\" (UniqueName: \"kubernetes.io/projected/1d4473c0-57b5-44a8-a467-d6a29d958c68-kube-api-access-jzlmm\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939041    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d4473c0-57b5-44a8-a467-d6a29d958c68-config-volume\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939092    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939133    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c63d965-5a4f-481a-8b25-0ac731629615-hosts-file\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939160    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-node-log\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939186    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939198    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-multus\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939228    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-dbus\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939231    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-node-log\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939248    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c63d965-5a4f-481a-8b25-0ac731629615-hosts-file\") pod \"node-resolver-q22m7\" (UID: \"8c63d965-5a4f-481a-8b25-0ac731629615\") " pod="openshift-dns/node-resolver-q22m7"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939251    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-ovn\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939268    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1697fe5f-c2b8-46e3-b57d-73afd75e6926-host-var-lib-cni-multus\") pod \"multus-vbrfn\" (UID: \"1697fe5f-c2b8-46e3-b57d-73afd75e6926\") " pod="openshift-multus/multus-vbrfn"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939281    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-ovn\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939299    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939333    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysconfig\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939371    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-kubernetes\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939433    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-key\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939483    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-cabundle\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939507    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-socket-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939532    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-registration-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939557    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939582    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-netd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939636    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-registration-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939644    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c7af65-aff0-41b1-8f98-2adf11d53954-socket-dir\") pod \"aws-ebs-csi-driver-node-s8fc2\" (UID: \"44c7af65-aff0-41b1-8f98-2adf11d53954\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8fc2"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939643    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx2mr\" (UniqueName: \"kubernetes.io/projected/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-api-access-lx2mr\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939673    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-cabundle\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939680    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bbbb1119-eb4e-48f1-8d3c-2c9264080717-signing-key\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:27.939686    4531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939722    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-host-cni-netd\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.939687    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1-run-openvswitch\") pod \"ovnkube-node-dqz7h\" (UID: \"4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:27.940490 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:27.939760    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs podName:448841dc-bb08-4989-8180-abe966bcaab4 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.4397435 +0000 UTC m=+1.709866777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs") pod "network-metrics-daemon-mgl8h" (UID: "448841dc-bb08-4989-8180-abe966bcaab4") : secret "metrics-daemon-secret" not found
Nov 03 19:31:27.970965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.970920    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlmm\" (UniqueName: \"kubernetes.io/projected/1d4473c0-57b5-44a8-a467-d6a29d958c68-kube-api-access-jzlmm\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:27.970965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.970949    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx2mr\" (UniqueName: \"kubernetes.io/projected/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-api-access-lx2mr\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:27.971153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.970949    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5nb\" (UniqueName: \"kubernetes.io/projected/ab89d762-778b-4e85-b506-72be48727030-kube-api-access-hz5nb\") pod \"migrator-66df5d496b-fk9b4\" (UID: \"ab89d762-778b-4e85-b506-72be48727030\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fk9b4"
Nov 03 19:31:27.971218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.971189    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p9qg\" (UniqueName: \"kubernetes.io/projected/99d9bea7-e81a-4fd7-8757-1eadfe124622-kube-api-access-2p9qg\") pod \"ingress-canary-8pptd\" (UID: \"99d9bea7-e81a-4fd7-8757-1eadfe124622\") " pod="openshift-ingress-canary/ingress-canary-8pptd"
Nov 03 19:31:27.971308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.971292    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwvm6\" (UniqueName: \"kubernetes.io/projected/bbbb1119-eb4e-48f1-8d3c-2c9264080717-kube-api-access-nwvm6\") pod \"service-ca-5fffc97bdf-pdfb6\" (UID: \"bbbb1119-eb4e-48f1-8d3c-2c9264080717\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pdfb6"
Nov 03 19:31:27.971380 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.971364    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtjc\" (UniqueName: \"kubernetes.io/projected/c455037a-fc47-4c3d-a917-854f924a5df0-kube-api-access-frtjc\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:27.972098 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:27.972077    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xng5d\" (UniqueName: \"kubernetes.io/projected/ba2d3316-f9d0-4812-89a3-d34efcc04628-kube-api-access-xng5d\") pod \"periodic-gathering-9xxl7-qt48t\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") " pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:31:28.025576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.025542    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:28.040701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040579    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-var-lib-kubelet\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.040701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040622    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-run\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.040701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040651    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-sys\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.040701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040676    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-conf\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.040701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040704    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040722    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9abeb54a-e625-4ad1-827e-49f5640d6288-crio-socket\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040740    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-modprobe-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040763    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-lib-modules\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040786    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-host\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040787    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-sys\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040786    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-run\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040818    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040820    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9abeb54a-e625-4ad1-827e-49f5640d6288-crio-socket\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:28.040845    4531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040858    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-host\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040871    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-kubelet-config\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040875    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-var-lib-kubelet\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:28.040906    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls podName:1d4473c0-57b5-44a8-a467-d6a29d958c68 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.540887304 +0000 UTC m=+1.811010606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls") pod "dns-default-tzpld" (UID: "1d4473c0-57b5-44a8-a467-d6a29d958c68") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040907    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-lib-modules\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040888    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-conf\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040926    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-kubelet-config\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040937    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040992    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-modprobe-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.040953    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysctl-d\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.041020    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-dbus\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.041055    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:28.041052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.041066    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/319a92b5-37b5-40b2-88cd-5550aab72650-dbus\") pod \"global-pull-secret-syncer-vj6pd\" (UID: \"319a92b5-37b5-40b2-88cd-5550aab72650\") " pod="kube-system/global-pull-secret-syncer-vj6pd"
Nov 03 19:31:28.041690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.041078    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysconfig\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.041102    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-kubernetes\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.041117    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-sysconfig\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.041133    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-systemd\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.041162    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-kubernetes\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041690 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:28.041194    4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:28.041690 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:28.041243    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls podName:9abeb54a-e625-4ad1-827e-49f5640d6288 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.541231826 +0000 UTC m=+1.811355098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjzpc" (UID: "9abeb54a-e625-4ad1-827e-49f5640d6288") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:28.041690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.041200    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c455037a-fc47-4c3d-a917-854f924a5df0-etc-systemd\") pod \"tuned-mfjxm\" (UID: \"c455037a-fc47-4c3d-a917-854f924a5df0\") " pod="openshift-cluster-node-tuning-operator/tuned-mfjxm"
Nov 03 19:31:28.041690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.041445    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9abeb54a-e625-4ad1-827e-49f5640d6288-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:28.078963 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.078897    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q22m7_8c63d965-5a4f-481a-8b25-0ac731629615/dns-node-resolver/0.log"
Nov 03 19:31:28.235499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.235443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:28.404062 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.404030    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:28.443569 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.443526    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:28.443735 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:28.443660    4531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 19:31:28.443822 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:28.443740    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs podName:448841dc-bb08-4989-8180-abe966bcaab4 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.443720857 +0000 UTC m=+2.713844147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs") pod "network-metrics-daemon-mgl8h" (UID: "448841dc-bb08-4989-8180-abe966bcaab4") : secret "metrics-daemon-secret" not found
Nov 03 19:31:28.532653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.532618    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:28.544750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.544714    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:28.544894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.544767    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:28.544894 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:28.544858    4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:28.544894 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:28.544871    4531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:28.545075 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:28.544922    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls podName:9abeb54a-e625-4ad1-827e-49f5640d6288 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.544908801 +0000 UTC m=+2.815032074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjzpc" (UID: "9abeb54a-e625-4ad1-827e-49f5640d6288") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:28.545075 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:28.544937    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls podName:1d4473c0-57b5-44a8-a467-d6a29d958c68 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.544928757 +0000 UTC m=+2.815052030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls") pod "dns-default-tzpld" (UID: "1d4473c0-57b5-44a8-a467-d6a29d958c68") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:28.678446 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:28.678374    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gcvlj_adabc7a3-99a6-43f0-b1ed-8a0395d310a4/node-ca/0.log"
Nov 03 19:31:29.213535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:29.213500    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:29.450840 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:29.450801    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:29.451014 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:29.450954    4531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 19:31:29.451051 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:29.451019    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs podName:448841dc-bb08-4989-8180-abe966bcaab4 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:31.451004274 +0000 UTC m=+4.721127547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs") pod "network-metrics-daemon-mgl8h" (UID: "448841dc-bb08-4989-8180-abe966bcaab4") : secret "metrics-daemon-secret" not found
Nov 03 19:31:29.478968 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:29.478892    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8pptd_99d9bea7-e81a-4fd7-8757-1eadfe124622/serve-healthcheck-canary/0.log"
Nov 03 19:31:29.551640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:29.551598    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:29.551801 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:29.551663    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:29.551801 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:29.551759    4531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:29.551901 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:29.551824    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls podName:1d4473c0-57b5-44a8-a467-d6a29d958c68 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:31.551808657 +0000 UTC m=+4.821931932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls") pod "dns-default-tzpld" (UID: "1d4473c0-57b5-44a8-a467-d6a29d958c68") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:29.551901 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:29.551765    4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:29.551901 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:29.551857    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls podName:9abeb54a-e625-4ad1-827e-49f5640d6288 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:31.551851523 +0000 UTC m=+4.821974796 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjzpc" (UID: "9abeb54a-e625-4ad1-827e-49f5640d6288") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:29.605929 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:29.605894    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:29.803814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:29.803725    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:30.006306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:30.006274    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:30.079334 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:30.079259    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fk9b4_ab89d762-778b-4e85-b506-72be48727030/migrator/0.log"
Nov 03 19:31:30.391864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:30.391833    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:31.213643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.213611    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:31.341535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.341504    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:31.463752 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.463677    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:31.464119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.463922    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:31.464307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.464289    4531 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:31:31.466328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.466303    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448841dc-bb08-4989-8180-abe966bcaab4-metrics-certs\") pod \"network-metrics-daemon-mgl8h\" (UID: \"448841dc-bb08-4989-8180-abe966bcaab4\") " pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:31.565134 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.565093    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:31.565332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.565161    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:31.565332 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:31.565286    4531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:31.565402 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:31.565345    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls podName:1d4473c0-57b5-44a8-a467-d6a29d958c68 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:35.565330643 +0000 UTC m=+8.835453920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls") pod "dns-default-tzpld" (UID: "1d4473c0-57b5-44a8-a467-d6a29d958c68") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:31.567548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.567519    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9abeb54a-e625-4ad1-827e-49f5640d6288-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjzpc\" (UID: \"9abeb54a-e625-4ad1-827e-49f5640d6288\") " pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:31.603244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.603214    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:31.712511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.712463    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgl8h"
Nov 03 19:31:31.727380 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.727353    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rjzpc"
Nov 03 19:31:31.812912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.812881    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:31.993377 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.993306    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgl8h"]
Nov 03 19:31:31.997117 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:31.997090    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448841dc_bb08_4989_8180_abe966bcaab4.slice/crio-0fa315505deb298f914dc516e98f69b14e6987e046d40140647a208aabcd26c8 WatchSource:0}: Error finding container 0fa315505deb298f914dc516e98f69b14e6987e046d40140647a208aabcd26c8: Status 404 returned error can't find the container with id 0fa315505deb298f914dc516e98f69b14e6987e046d40140647a208aabcd26c8
Nov 03 19:31:31.999075 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:31.999059    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:31:32.003865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:32.003839    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rjzpc"]
Nov 03 19:31:32.007305 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:32.007281    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abeb54a_e625_4ad1_827e_49f5640d6288.slice/crio-b32b2edd531ae0b3b852f0079f28a80aa1f6f9c9dc2c09412097ead19f5f8e2c WatchSource:0}: Error finding container b32b2edd531ae0b3b852f0079f28a80aa1f6f9c9dc2c09412097ead19f5f8e2c: Status 404 returned error can't find the container with id b32b2edd531ae0b3b852f0079f28a80aa1f6f9c9dc2c09412097ead19f5f8e2c
Nov 03 19:31:32.020150 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:32.020117    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:32.201877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:32.201844    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:31:32.210026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:32.209985    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:32.919325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:32.919283    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgl8h" event={"ID":"448841dc-bb08-4989-8180-abe966bcaab4","Type":"ContainerStarted","Data":"0fa315505deb298f914dc516e98f69b14e6987e046d40140647a208aabcd26c8"}
Nov 03 19:31:32.921350 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:32.921318    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjzpc" event={"ID":"9abeb54a-e625-4ad1-827e-49f5640d6288","Type":"ContainerStarted","Data":"947bbb3ff3367a8a469f5dbebe06e3bf6d3c2783ca3a623f95c00141d9debe61"}
Nov 03 19:31:32.921498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:32.921375    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjzpc" event={"ID":"9abeb54a-e625-4ad1-827e-49f5640d6288","Type":"ContainerStarted","Data":"0e14e7001276d5291475d5cb98232ffda38edab6f197d4febb1f22724d3ffdc5"}
Nov 03 19:31:32.921498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:32.921406    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjzpc" event={"ID":"9abeb54a-e625-4ad1-827e-49f5640d6288","Type":"ContainerStarted","Data":"b32b2edd531ae0b3b852f0079f28a80aa1f6f9c9dc2c09412097ead19f5f8e2c"}
Nov 03 19:31:33.265569 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:33.265533    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:33.401058 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:33.401021    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:33.606038 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:33.605995    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:33.800794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:33.800722    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:33.927768 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:33.927728    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgl8h" event={"ID":"448841dc-bb08-4989-8180-abe966bcaab4","Type":"ContainerStarted","Data":"525b33ad3167f81305a34b10a7469f846799185fe279d277be32fdfb1dc9439a"}
Nov 03 19:31:33.928150 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:33.927792    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgl8h" event={"ID":"448841dc-bb08-4989-8180-abe966bcaab4","Type":"ContainerStarted","Data":"c8f81f2fd54d87c640372dba5e6760af7afceef8658028318185893b969be5ae"}
Nov 03 19:31:34.005136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:34.005100    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:34.205201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:34.205163    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:34.932917 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:34.932882    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjzpc" event={"ID":"9abeb54a-e625-4ad1-827e-49f5640d6288","Type":"ContainerStarted","Data":"63e481f3d22bfce18017e26e0182f46b26b15486b7e8e92020fb86d700c0d894"}
Nov 03 19:31:35.237491 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:35.237421    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:35.407487 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:35.407438    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:35.593521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:35.593432    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:35.595813 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:35.595791    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d4473c0-57b5-44a8-a467-d6a29d958c68-metrics-tls\") pod \"dns-default-tzpld\" (UID: \"1d4473c0-57b5-44a8-a467-d6a29d958c68\") " pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:35.608019 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:35.607987    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:35.623990 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:35.623955    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:35.806359 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:35.806329    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:35.890217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:35.890186    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tzpld"]
Nov 03 19:31:35.893659 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:35.893631    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4473c0_57b5_44a8_a467_d6a29d958c68.slice/crio-510d261fe94315f618cb51fb27f1ab29af68b9e7c34a7110375e676b0ed94787 WatchSource:0}: Error finding container 510d261fe94315f618cb51fb27f1ab29af68b9e7c34a7110375e676b0ed94787: Status 404 returned error can't find the container with id 510d261fe94315f618cb51fb27f1ab29af68b9e7c34a7110375e676b0ed94787
Nov 03 19:31:35.937514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:35.937481    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tzpld" event={"ID":"1d4473c0-57b5-44a8-a467-d6a29d958c68","Type":"ContainerStarted","Data":"510d261fe94315f618cb51fb27f1ab29af68b9e7c34a7110375e676b0ed94787"}
Nov 03 19:31:36.022960 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:36.022928    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:36.169522 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:36.169429    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:36.291803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:36.291768    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:37.545042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:37.545009    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:37.545504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:37.545263    4531 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:31:38.145789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:38.145755    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:38.274787 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:38.274700    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:38.400405 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:38.400370    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:38.531239 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:38.531147    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:38.949140 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:38.949103    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tzpld" event={"ID":"1d4473c0-57b5-44a8-a467-d6a29d958c68","Type":"ContainerStarted","Data":"efd94ae66995842bc83c3ab1d4519d2be0e6f2afedde3c0759ec41024f02a0c5"}
Nov 03 19:31:38.949545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:38.949163    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tzpld" event={"ID":"1d4473c0-57b5-44a8-a467-d6a29d958c68","Type":"ContainerStarted","Data":"295dc4b38f62a2ec4552f865f5b5681cd71a762fe70695e333d7ba683f75b860"}
Nov 03 19:31:38.949545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:38.949297    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:39.573219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.573163    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:39.786424 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.786388    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jj9xh-5rsck"]
Nov 03 19:31:39.788401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.788374    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:39.789572 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.789552    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.805669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.805482    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jj9xh-5rsck"]
Nov 03 19:31:39.822111 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.822081    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7a924521-7745-40c7-b46d-1a9e5b3375ea-archives-path\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.822292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.822120    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn62c\" (UniqueName: \"kubernetes.io/projected/7a924521-7745-40c7-b46d-1a9e5b3375ea-kube-api-access-jn62c\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.822292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.822232    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-service-ca-bundle\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.822292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.822250    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-trusted-ca-bundle\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.922983 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.922944    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-service-ca-bundle\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.922983 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.922981    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-trusted-ca-bundle\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.923271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.923007    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7a924521-7745-40c7-b46d-1a9e5b3375ea-archives-path\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.923271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.923025    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn62c\" (UniqueName: \"kubernetes.io/projected/7a924521-7745-40c7-b46d-1a9e5b3375ea-kube-api-access-jn62c\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.923541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.923517    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7a924521-7745-40c7-b46d-1a9e5b3375ea-archives-path\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.923850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.923826    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-service-ca-bundle\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.924158 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.924140    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-trusted-ca-bundle\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.934821 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.934796    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn62c\" (UniqueName: \"kubernetes.io/projected/7a924521-7745-40c7-b46d-1a9e5b3375ea-kube-api-access-jn62c\") pod \"periodic-gathering-jj9xh-5rsck\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") " pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:39.950365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:39.950337    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:40.100795 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:40.100755    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:31:40.101877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:40.101848    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:40.237357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:40.237260    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:40.369107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:40.369076    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jj9xh-5rsck"]
Nov 03 19:31:40.372495 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:40.372461    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a924521_7745_40c7_b46d_1a9e5b3375ea.slice/crio-67abb9ba30b5b88c5de77d07fa9e211a53bd4ce966ae475e34e26b30722dedd8 WatchSource:0}: Error finding container 67abb9ba30b5b88c5de77d07fa9e211a53bd4ce966ae475e34e26b30722dedd8: Status 404 returned error can't find the container with id 67abb9ba30b5b88c5de77d07fa9e211a53bd4ce966ae475e34e26b30722dedd8
Nov 03 19:31:40.383729 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:40.383702    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:40.533677 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:40.533604    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:40.956971 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:40.956927    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jj9xh-5rsck" event={"ID":"7a924521-7745-40c7-b46d-1a9e5b3375ea","Type":"ContainerStarted","Data":"56b10afa66abd9826bf965fcfe98860afa442188b18cedb706236b5e431cf772"}
Nov 03 19:31:40.957396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:40.956989    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jj9xh-5rsck" event={"ID":"7a924521-7745-40c7-b46d-1a9e5b3375ea","Type":"ContainerStarted","Data":"67abb9ba30b5b88c5de77d07fa9e211a53bd4ce966ae475e34e26b30722dedd8"}
Nov 03 19:31:40.973259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:40.973205    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jj9xh-5rsck" podStartSLOduration=1.9731670289999999 podStartE2EDuration="1.973167029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:40.97220299 +0000 UTC m=+14.242326284" watchObservedRunningTime="2026-11-03 19:31:40.973167029 +0000 UTC m=+14.243290323"
Nov 03 19:31:41.636247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:41.636219    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dqz7h"
Nov 03 19:31:41.847655 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:41.847624    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:41.985638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:41.985561    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:42.112929 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:42.112894    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:42.243562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:42.243479    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:43.367373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.367339    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"]
Nov 03 19:31:43.370668 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.370652    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.376804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.376466    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zrbjf\""
Nov 03 19:31:43.376804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.376487    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:31:43.376804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.376513    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 19:31:43.376804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.376538    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:31:43.376804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.376618    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 19:31:43.377448 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.377162    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:31:43.377867 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.377844    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"]
Nov 03 19:31:43.433719 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.433684    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:43.449890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.449860    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f44c80-7900-4ec5-ba15-50cc0e236ca7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.449994 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.449914    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/43f44c80-7900-4ec5-ba15-50cc0e236ca7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.449994 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.449947    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-462wd\" (UniqueName: \"kubernetes.io/projected/43f44c80-7900-4ec5-ba15-50cc0e236ca7-kube-api-access-462wd\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.450084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.450026    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43f44c80-7900-4ec5-ba15-50cc0e236ca7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.551387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.551343    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/43f44c80-7900-4ec5-ba15-50cc0e236ca7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.551586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.551399    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-462wd\" (UniqueName: \"kubernetes.io/projected/43f44c80-7900-4ec5-ba15-50cc0e236ca7-kube-api-access-462wd\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.551586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.551441    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43f44c80-7900-4ec5-ba15-50cc0e236ca7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.551586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.551490    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f44c80-7900-4ec5-ba15-50cc0e236ca7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.552295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.552261    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f44c80-7900-4ec5-ba15-50cc0e236ca7-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.553863 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.553831    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43f44c80-7900-4ec5-ba15-50cc0e236ca7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.553952 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.553835    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/43f44c80-7900-4ec5-ba15-50cc0e236ca7-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.559589 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.559559    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-462wd\" (UniqueName: \"kubernetes.io/projected/43f44c80-7900-4ec5-ba15-50cc0e236ca7-kube-api-access-462wd\") pod \"prometheus-operator-9cd89b74c-d5rvl\" (UID: \"43f44c80-7900-4ec5-ba15-50cc0e236ca7\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.587978 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.587934    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:43.684796 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.684759    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"
Nov 03 19:31:43.957541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.957394    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl"]
Nov 03 19:31:43.961142 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:43.961111    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f44c80_7900_4ec5_ba15_50cc0e236ca7.slice/crio-30356fc77e4b6adf61f576c99b7ae98f01691cb5c04c32040f922d7cc31b4940 WatchSource:0}: Error finding container 30356fc77e4b6adf61f576c99b7ae98f01691cb5c04c32040f922d7cc31b4940: Status 404 returned error can't find the container with id 30356fc77e4b6adf61f576c99b7ae98f01691cb5c04c32040f922d7cc31b4940
Nov 03 19:31:43.967735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:43.967712    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl" event={"ID":"43f44c80-7900-4ec5-ba15-50cc0e236ca7","Type":"ContainerStarted","Data":"30356fc77e4b6adf61f576c99b7ae98f01691cb5c04c32040f922d7cc31b4940"}
Nov 03 19:31:44.130500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:44.130472    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tzpld_1d4473c0-57b5-44a8-a467-d6a29d958c68/dns/0.log"
Nov 03 19:31:44.330694 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:44.330619    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tzpld_1d4473c0-57b5-44a8-a467-d6a29d958c68/kube-rbac-proxy/0.log"
Nov 03 19:31:44.471436 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:44.471399    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4jl9f"
Nov 03 19:31:45.331022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:45.330987    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q22m7_8c63d965-5a4f-481a-8b25-0ac731629615/dns-node-resolver/0.log"
Nov 03 19:31:45.857806 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:45.857723    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:45.977057 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:45.977021    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl" event={"ID":"43f44c80-7900-4ec5-ba15-50cc0e236ca7","Type":"ContainerStarted","Data":"3e8ce4acee460c060270f72d8507d39b6d7546877cc00795c6230d74563c5ca8"}
Nov 03 19:31:45.977241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:45.977072    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl" event={"ID":"43f44c80-7900-4ec5-ba15-50cc0e236ca7","Type":"ContainerStarted","Data":"823e3c5a461d38f71c64c3f82e1556afe5e29d8136e1ac6d62fcbac17949b920"}
Nov 03 19:31:45.991190 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:45.991126    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-d5rvl" podStartSLOduration=1.478595049 podStartE2EDuration="2.991112454s" totalImagesPullingTime="1.512517405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:45.990533566 +0000 UTC m=+19.260656862" watchObservedRunningTime="2026-11-03 19:31:45.991112454 +0000 UTC m=+19.261235746"
Nov 03 19:31:46.130577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:46.130548    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gcvlj_adabc7a3-99a6-43f0-b1ed-8a0395d310a4/node-ca/0.log"
Nov 03 19:31:46.930779 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:46.930750    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8pptd_99d9bea7-e81a-4fd7-8757-1eadfe124622/serve-healthcheck-canary/0.log"
Nov 03 19:31:47.533600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.533563    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:47.681816 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.681778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:47.736840 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.736808    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tvvhh"]
Nov 03 19:31:47.740191 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.740158    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.751394 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.751365    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:31:47.751573 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.751412    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:31:47.751573 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.751459    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zffgv\""
Nov 03 19:31:47.780899 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.780864    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.780899 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.780901    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-tls\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.781123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.780930    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-wtmp\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.781123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.780983    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6bb340f1-f611-44c2-99e2-550c4c49e1b0-root\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.781123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.781082    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-textfile\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.781123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.781109    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bb340f1-f611-44c2-99e2-550c4c49e1b0-sys\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.781344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.781136    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xbsr\" (UniqueName: \"kubernetes.io/projected/6bb340f1-f611-44c2-99e2-550c4c49e1b0-kube-api-access-2xbsr\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.781344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.781151    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bb340f1-f611-44c2-99e2-550c4c49e1b0-metrics-client-ca\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.781344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.781231    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.816007 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.815923    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:31:47.827687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.827642    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:47.882099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882062    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bb340f1-f611-44c2-99e2-550c4c49e1b0-sys\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882109    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xbsr\" (UniqueName: \"kubernetes.io/projected/6bb340f1-f611-44c2-99e2-550c4c49e1b0-kube-api-access-2xbsr\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882129    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bb340f1-f611-44c2-99e2-550c4c49e1b0-metrics-client-ca\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882148    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882206    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882206    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bb340f1-f611-44c2-99e2-550c4c49e1b0-sys\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882226    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-tls\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882270    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-wtmp\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882303    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6bb340f1-f611-44c2-99e2-550c4c49e1b0-root\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882341    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-textfile\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882393    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6bb340f1-f611-44c2-99e2-550c4c49e1b0-root\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882458    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-wtmp\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882776    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-textfile\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882804    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bb340f1-f611-44c2-99e2-550c4c49e1b0-metrics-client-ca\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.882849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.882815    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.884601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.884580    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-tls\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.884697 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.884669    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bb340f1-f611-44c2-99e2-550c4c49e1b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.889898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.889871    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xbsr\" (UniqueName: \"kubernetes.io/projected/6bb340f1-f611-44c2-99e2-550c4c49e1b0-kube-api-access-2xbsr\") pod \"node-exporter-tvvhh\" (UID: \"6bb340f1-f611-44c2-99e2-550c4c49e1b0\") " pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:47.959944 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:47.959911    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:48.055957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:48.055920    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tvvhh"
Nov 03 19:31:48.067723 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:48.067662    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb340f1_f611_44c2_99e2_550c4c49e1b0.slice/crio-d3754290c53166251f7165054b0022c7c57ba51e2fbf79bc1b476bb883d8fc1a WatchSource:0}: Error finding container d3754290c53166251f7165054b0022c7c57ba51e2fbf79bc1b476bb883d8fc1a: Status 404 returned error can't find the container with id d3754290c53166251f7165054b0022c7c57ba51e2fbf79bc1b476bb883d8fc1a
Nov 03 19:31:48.138531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:48.138482    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:48.271850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:48.271815    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:48.767585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:48.767528    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:48.899416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:48.899374    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:48.954325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:48.954295    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tzpld"
Nov 03 19:31:48.994533 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:48.994492    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvvhh" event={"ID":"6bb340f1-f611-44c2-99e2-550c4c49e1b0","Type":"ContainerStarted","Data":"d3754290c53166251f7165054b0022c7c57ba51e2fbf79bc1b476bb883d8fc1a"}
Nov 03 19:31:49.038776 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.038688    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:49.167451 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.167414    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:49.298986 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.298843    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:49.689428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.689392    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bf5978794-gs47k"]
Nov 03 19:31:49.692827 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.692807    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.695816 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.695789    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 19:31:49.695919 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.695830    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 19:31:49.695919 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.695872    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 19:31:49.696272 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.696258    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 19:31:49.696339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.696261    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5xzdf\""
Nov 03 19:31:49.696378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.696361    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4j21kbervmojl\""
Nov 03 19:31:49.696423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.696379    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 19:31:49.703593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.703501    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bf5978794-gs47k"]
Nov 03 19:31:49.796916 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.796874    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.797108 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.796922    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.797108 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.796999    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-grpc-tls\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.797215 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.797099    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.797215 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.797140    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-tmp\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.797280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.797221    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-metrics-client-ca\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.797280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.797250    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-tls\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.797368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.797283    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.797368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.797313    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spr5k\" (UniqueName: \"kubernetes.io/projected/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-kube-api-access-spr5k\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.898029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.897987    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-grpc-tls\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.898029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.898034    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.898335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.898055    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-tmp\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.898335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.898079    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-metrics-client-ca\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.898335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.898112    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-tls\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.898335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.898151    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.898335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.898197    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spr5k\" (UniqueName: \"kubernetes.io/projected/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-kube-api-access-spr5k\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.898335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.898255    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.898335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.898288    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.898709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.898587    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-tmp\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.899021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.898971    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-metrics-client-ca\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.900878 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.900852    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.900985 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.900852    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.901140 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.901121    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-grpc-tls\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.901224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.901163    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-tls\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.901224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.901193    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.901300 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.901233    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.905814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.905790    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spr5k\" (UniqueName: \"kubernetes.io/projected/b2f2b56d-0c94-496e-a57f-8d83a7df23cb-kube-api-access-spr5k\") pod \"thanos-querier-6bf5978794-gs47k\" (UID: \"b2f2b56d-0c94-496e-a57f-8d83a7df23cb\") " pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:49.999270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.999167    4531 generic.go:356] "Generic (PLEG): container finished" podID="6bb340f1-f611-44c2-99e2-550c4c49e1b0" containerID="d8ed122f2af2d6e4f9b899898f6cb7d7802cc4517202a7d65d49f9bcfb7c5661" exitCode=0
Nov 03 19:31:49.999270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:49.999249    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvvhh" event={"ID":"6bb340f1-f611-44c2-99e2-550c4c49e1b0","Type":"ContainerDied","Data":"d8ed122f2af2d6e4f9b899898f6cb7d7802cc4517202a7d65d49f9bcfb7c5661"}
Nov 03 19:31:50.017572 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:50.017531    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:50.293015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:50.292934    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bf5978794-gs47k"]
Nov 03 19:31:50.295950 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:50.295909    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f2b56d_0c94_496e_a57f_8d83a7df23cb.slice/crio-5bc1b3065792565297e216fe294c5169d988f01c53e02a516ef5b4e384650c34 WatchSource:0}: Error finding container 5bc1b3065792565297e216fe294c5169d988f01c53e02a516ef5b4e384650c34: Status 404 returned error can't find the container with id 5bc1b3065792565297e216fe294c5169d988f01c53e02a516ef5b4e384650c34
Nov 03 19:31:50.572492 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:31:50.572415    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:31:50.700326 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:50.700293    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:50.904886 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:50.904853    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:51.005499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:51.005463    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvvhh" event={"ID":"6bb340f1-f611-44c2-99e2-550c4c49e1b0","Type":"ContainerStarted","Data":"b1b0eea8bb36c3d485279abbae6eb1daf561bce60cc88c2ae8efd73fe3e01e86"}
Nov 03 19:31:51.005939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:51.005527    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvvhh" event={"ID":"6bb340f1-f611-44c2-99e2-550c4c49e1b0","Type":"ContainerStarted","Data":"40c108faadf1a6d1dc24d52eda4ed41e87a1ca6c53e42aae3677a0efe2cc96c3"}
Nov 03 19:31:51.006724 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:51.006696    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k" event={"ID":"b2f2b56d-0c94-496e-a57f-8d83a7df23cb","Type":"ContainerStarted","Data":"5bc1b3065792565297e216fe294c5169d988f01c53e02a516ef5b4e384650c34"}
Nov 03 19:31:51.021607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:51.021550    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tvvhh" podStartSLOduration=2.773804411 podStartE2EDuration="4.021532525s" totalImagesPullingTime="1.180615373s" totalInitContainerRuntime="67.112741ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:51.021046163 +0000 UTC m=+24.291169460" watchObservedRunningTime="2026-11-03 19:31:51.021532525 +0000 UTC m=+24.291655820"
Nov 03 19:31:52.099608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.099575    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84bd6f669d-jvbjq"]
Nov 03 19:31:52.103049 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.103016    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.103424 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.103395    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:52.112694 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.112667    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 19:31:52.112830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.112695    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 19:31:52.112830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.112780    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84bd6f669d-jvbjq"]
Nov 03 19:31:52.150231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.150197    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 19:31:52.150428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.150355    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 19:31:52.150428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.150384    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jtfds\""
Nov 03 19:31:52.150552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.150458    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7isue58tnoudh\""
Nov 03 19:31:52.216080 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.216032    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7668ec48-a5a4-4912-b786-604de5458489-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.216080 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.216079    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7668ec48-a5a4-4912-b786-604de5458489-metrics-server-audit-profiles\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.216349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.216104    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7668ec48-a5a4-4912-b786-604de5458489-secret-metrics-server-client-certs\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.216349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.216275    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7668ec48-a5a4-4912-b786-604de5458489-secret-metrics-server-tls\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.216349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.216331    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7rz\" (UniqueName: \"kubernetes.io/projected/7668ec48-a5a4-4912-b786-604de5458489-kube-api-access-jn7rz\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.216509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.216362    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7668ec48-a5a4-4912-b786-604de5458489-audit-log\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.216509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.216384    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7668ec48-a5a4-4912-b786-604de5458489-client-ca-bundle\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.237744 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.237708    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:52.317187 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.317151    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7668ec48-a5a4-4912-b786-604de5458489-secret-metrics-server-tls\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.317294 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.317211    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7rz\" (UniqueName: \"kubernetes.io/projected/7668ec48-a5a4-4912-b786-604de5458489-kube-api-access-jn7rz\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.317294 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.317238    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7668ec48-a5a4-4912-b786-604de5458489-audit-log\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.317294 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.317264    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7668ec48-a5a4-4912-b786-604de5458489-client-ca-bundle\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.317451 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.317322    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7668ec48-a5a4-4912-b786-604de5458489-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.317451 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.317345    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7668ec48-a5a4-4912-b786-604de5458489-metrics-server-audit-profiles\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.317451 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.317372    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7668ec48-a5a4-4912-b786-604de5458489-secret-metrics-server-client-certs\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.318077 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.317999    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7668ec48-a5a4-4912-b786-604de5458489-audit-log\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.318283 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.318254    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7668ec48-a5a4-4912-b786-604de5458489-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.318490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.318469    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7668ec48-a5a4-4912-b786-604de5458489-metrics-server-audit-profiles\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.320623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.320601    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7668ec48-a5a4-4912-b786-604de5458489-secret-metrics-server-tls\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.320713 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.320620    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7668ec48-a5a4-4912-b786-604de5458489-client-ca-bundle\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.320801 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.320782    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7668ec48-a5a4-4912-b786-604de5458489-secret-metrics-server-client-certs\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.323827 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.323808    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7rz\" (UniqueName: \"kubernetes.io/projected/7668ec48-a5a4-4912-b786-604de5458489-kube-api-access-jn7rz\") pod \"metrics-server-84bd6f669d-jvbjq\" (UID: \"7668ec48-a5a4-4912-b786-604de5458489\") " pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.416106 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.416076    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:31:52.575979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.575948    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:52.697573 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.697489    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84bd6f669d-jvbjq"]
Nov 03 19:31:52.701689 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.701663    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:52.701810 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:52.701791    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7668ec48_a5a4_4912_b786_604de5458489.slice/crio-95d43e02a567ed1ec09126274d2c67bb08f23a86559e22e41665e8d56657dfad WatchSource:0}: Error finding container 95d43e02a567ed1ec09126274d2c67bb08f23a86559e22e41665e8d56657dfad: Status 404 returned error can't find the container with id 95d43e02a567ed1ec09126274d2c67bb08f23a86559e22e41665e8d56657dfad
Nov 03 19:31:52.903052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.903015    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"]
Nov 03 19:31:52.906718 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.906698    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:52.911008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.910972    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 19:31:52.911383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.911346    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 19:31:52.911531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.911471    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 19:31:52.911700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.911683    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 19:31:52.911785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.911732    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 19:31:52.911785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.911753    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bq4pq\""
Nov 03 19:31:52.912594 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.912574    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 19:31:52.917672 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:52.917650    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"]
Nov 03 19:31:53.015889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.015794    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k" event={"ID":"b2f2b56d-0c94-496e-a57f-8d83a7df23cb","Type":"ContainerStarted","Data":"420afe7481e8aa98ec28562411bec2548afd32a6a3a4c910bc2a67b69566a9fc"}
Nov 03 19:31:53.015889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.015884    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k" event={"ID":"b2f2b56d-0c94-496e-a57f-8d83a7df23cb","Type":"ContainerStarted","Data":"565f1eb1afd62a6f3103b3dec95602b7acc1450cbc67235db240be3b14883f6f"}
Nov 03 19:31:53.016087 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.015913    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k" event={"ID":"b2f2b56d-0c94-496e-a57f-8d83a7df23cb","Type":"ContainerStarted","Data":"8ebe80ef67bc35ff709b454ca0d921e0379aedbaf9c92334bc3b7b7a8249121a"}
Nov 03 19:31:53.016934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.016911    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq" event={"ID":"7668ec48-a5a4-4912-b786-604de5458489","Type":"ContainerStarted","Data":"95d43e02a567ed1ec09126274d2c67bb08f23a86559e22e41665e8d56657dfad"}
Nov 03 19:31:53.023314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.023287    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.023411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.023325    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-telemeter-client-tls\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.023411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.023350    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-secret-telemeter-client\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.023411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.023393    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-federate-client-tls\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.023527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.023428    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxklh\" (UniqueName: \"kubernetes.io/projected/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-kube-api-access-lxklh\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.023527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.023505    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-serving-certs-ca-bundle\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.023600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.023541    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.023600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.023571    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-metrics-client-ca\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.124359 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.124323    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-metrics-client-ca\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.124803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.124368    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.124803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.124392    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-telemeter-client-tls\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.124803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.124460    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-secret-telemeter-client\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.124803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.124486    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-federate-client-tls\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.124803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.124517    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxklh\" (UniqueName: \"kubernetes.io/projected/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-kube-api-access-lxklh\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.124803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.124623    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-serving-certs-ca-bundle\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.124803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.124673    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.125718 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.125628    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-metrics-client-ca\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.126453 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.126423    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-telemeter-trusted-ca-bundle\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.127995 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.127899    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-serving-certs-ca-bundle\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.128461 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.128436    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-secret-telemeter-client\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.132354 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.132321    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-federate-client-tls\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.132514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.132398    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxklh\" (UniqueName: \"kubernetes.io/projected/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-kube-api-access-lxklh\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.132829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.132805    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-telemeter-client-tls\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.133660 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.133633    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-9fbf4f6f9-8p6lk\" (UID: \"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135\") " pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.222715 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.222670    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"
Nov 03 19:31:53.510906 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.510878    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk"]
Nov 03 19:31:53.516454 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:53.516426    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2b56e8_2b00_4fdf_9ab4_e44e47cdc135.slice/crio-c325944afc6ed7483d855b4816c723c9e09223c319185bbee80f5f75c1c74dd9 WatchSource:0}: Error finding container c325944afc6ed7483d855b4816c723c9e09223c319185bbee80f5f75c1c74dd9: Status 404 returned error can't find the container with id c325944afc6ed7483d855b4816c723c9e09223c319185bbee80f5f75c1c74dd9
Nov 03 19:31:53.999068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:53.999015    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:54.026623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:54.026292    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k" event={"ID":"b2f2b56d-0c94-496e-a57f-8d83a7df23cb","Type":"ContainerStarted","Data":"536c7d53e3545c2f4ad2906ae01d0f0facdc00bd69e963118018af282c26ef12"}
Nov 03 19:31:54.026623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:54.026576    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:31:54.026865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:54.026647    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k" event={"ID":"b2f2b56d-0c94-496e-a57f-8d83a7df23cb","Type":"ContainerStarted","Data":"6db964bfab43a6e39f019fd08c53173cd0b09c862cdb9849a04ffc21b3b10165"}
Nov 03 19:31:54.026865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:54.026694    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k" event={"ID":"b2f2b56d-0c94-496e-a57f-8d83a7df23cb","Type":"ContainerStarted","Data":"27b8d915f52698900d59d35e659ac35aa98eeb054e011012822a0b5bb7e6610d"}
Nov 03 19:31:54.032022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:54.031993    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk" event={"ID":"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135","Type":"ContainerStarted","Data":"c325944afc6ed7483d855b4816c723c9e09223c319185bbee80f5f75c1c74dd9"}
Nov 03 19:31:54.045336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:54.045275    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k" podStartSLOduration=2.102172566 podStartE2EDuration="5.045254592s" totalImagesPullingTime="2.943082026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:54.045135003 +0000 UTC m=+27.315258311" watchObservedRunningTime="2026-11-03 19:31:54.045254592 +0000 UTC m=+27.315377867"
Nov 03 19:31:54.411864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:54.411825    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:54.536059 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:54.535966    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:55.037578 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:55.037538    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq" event={"ID":"7668ec48-a5a4-4912-b786-604de5458489","Type":"ContainerStarted","Data":"726bbed1de76c879ea38902016ed18d1bb93fa60bfff2687e7d5884c5bd58321"}
Nov 03 19:31:55.054020 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:55.053976    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq" podStartSLOduration=1.57283486 podStartE2EDuration="3.053959118s" totalImagesPullingTime="1.481124258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:55.053079519 +0000 UTC m=+28.323202814" watchObservedRunningTime="2026-11-03 19:31:55.053959118 +0000 UTC m=+28.324082413"
Nov 03 19:31:55.359068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:55.358978    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:55.481263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:55.481225    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:55.885753 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:55.885720    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:56.006136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:56.006102    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:56.043523 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:56.043491    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk" event={"ID":"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135","Type":"ContainerStarted","Data":"401c94472a7c069fb0d9630c5aa951e9ed0f1b0fa3915c7b51fd59b4bce5ee5a"}
Nov 03 19:31:57.224452 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.224421    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67d657d848-gvgc6"]
Nov 03 19:31:57.238344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.238318    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d657d848-gvgc6"]
Nov 03 19:31:57.238449 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.238424    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.242405 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.242335    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 19:31:57.242405 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.242342    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 19:31:57.242405 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.242351    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 19:31:57.242684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.242484    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 19:31:57.242684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.242486    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 19:31:57.242684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.242529    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 19:31:57.242840 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.242823    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 19:31:57.242918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.242901    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-46fdn\""
Nov 03 19:31:57.244238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.244220    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 19:31:57.359113 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.359013    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-oauth-config\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.359113 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.359066    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02690ad5-61e0-4357-aa9e-4d3530469ba1-tmp\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.359346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.359150    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-trusted-ca-bundle\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.359346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.359210    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4b6v\" (UniqueName: \"kubernetes.io/projected/02690ad5-61e0-4357-aa9e-4d3530469ba1-kube-api-access-t4b6v\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.359346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.359307    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-service-ca\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.359462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.359350    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-serving-cert\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.359462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.359371    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-config\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.359548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.359503    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-oauth-serving-cert\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.460378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.460341    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-service-ca\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.460541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.460390    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-serving-cert\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.460541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.460414    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-config\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.460541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.460466    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-oauth-serving-cert\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.460541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.460500    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-oauth-config\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.460541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.460531    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02690ad5-61e0-4357-aa9e-4d3530469ba1-tmp\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.460810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.460554    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-trusted-ca-bundle\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.460810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.460582    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4b6v\" (UniqueName: \"kubernetes.io/projected/02690ad5-61e0-4357-aa9e-4d3530469ba1-kube-api-access-t4b6v\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.461284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.461258    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-service-ca\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.461284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.461271    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02690ad5-61e0-4357-aa9e-4d3530469ba1-tmp\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.461420 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.461325    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-config\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.461530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.461507    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-oauth-serving-cert\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.462004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.461980    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-trusted-ca-bundle\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.463026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.463006    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-serving-cert\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.463163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.463147    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-oauth-config\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.467377 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.467355    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4b6v\" (UniqueName: \"kubernetes.io/projected/02690ad5-61e0-4357-aa9e-4d3530469ba1-kube-api-access-t4b6v\") pod \"console-67d657d848-gvgc6\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") " pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.570976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.570938    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:31:57.875653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:57.875579    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d657d848-gvgc6"]
Nov 03 19:31:57.878704 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:31:57.878676    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02690ad5_61e0_4357_aa9e_4d3530469ba1.slice/crio-4b7e0b8479638c9393691b348fee3375ae059d5869d1574327bf06ca42750730 WatchSource:0}: Error finding container 4b7e0b8479638c9393691b348fee3375ae059d5869d1574327bf06ca42750730: Status 404 returned error can't find the container with id 4b7e0b8479638c9393691b348fee3375ae059d5869d1574327bf06ca42750730
Nov 03 19:31:58.055465 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:58.055414    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk" event={"ID":"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135","Type":"ContainerStarted","Data":"ef8fd607ee5b1dea15b508768348775bfe0ee3d8af531a536434f6a1b90d42db"}
Nov 03 19:31:58.055662 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:58.055519    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk" event={"ID":"8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135","Type":"ContainerStarted","Data":"81e315a3b503767222c856806fbb4fa93fb3fcf91d78c6aa5327ac4cb80a5c31"}
Nov 03 19:31:58.057018 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:58.056991    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d657d848-gvgc6" event={"ID":"02690ad5-61e0-4357-aa9e-4d3530469ba1","Type":"ContainerStarted","Data":"4b7e0b8479638c9393691b348fee3375ae059d5869d1574327bf06ca42750730"}
Nov 03 19:31:58.079125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:58.079079    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-9fbf4f6f9-8p6lk" podStartSLOduration=2.6065873120000003 podStartE2EDuration="6.079064746s" totalImagesPullingTime="3.472477434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:58.078280049 +0000 UTC m=+31.348403346" watchObservedRunningTime="2026-11-03 19:31:58.079064746 +0000 UTC m=+31.349188041"
Nov 03 19:31:58.461810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:58.461774    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:58.603107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:58.603071    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:59.578342 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:31:59.578304    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:00.045265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:00.045240    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bf5978794-gs47k"
Nov 03 19:32:00.965933 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:00.965891    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:01.168073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:01.168045    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:02.073714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:02.073683    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d657d848-gvgc6" event={"ID":"02690ad5-61e0-4357-aa9e-4d3530469ba1","Type":"ContainerStarted","Data":"73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c"}
Nov 03 19:32:02.090351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:02.090249    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67d657d848-gvgc6" podStartSLOduration=1.844263249 podStartE2EDuration="5.090231479s" totalImagesPullingTime="3.24596823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:02.089919901 +0000 UTC m=+35.360043196" watchObservedRunningTime="2026-11-03 19:32:02.090231479 +0000 UTC m=+35.360354775"
Nov 03 19:32:02.207099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:02.207062    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m4rw5"
Nov 03 19:32:03.508375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:03.508335    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:03.635097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:03.635064    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:07.149522 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:07.149486    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:07.571937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:07.571857    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:32:07.571937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:07.571935    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:32:07.576613 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:07.576590    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:32:07.828547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:07.828458    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d657d848-gvgc6"]
Nov 03 19:32:08.105525 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:08.105453    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:32:09.240849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:09.240811    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:09.364684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:09.364653    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:12.416962 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.416924    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:32:12.417403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.417017    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:32:12.471685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.471647    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-9xxl7-qt48t"]
Nov 03 19:32:12.471869 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.471849    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-9xxl7-qt48t" podUID="ba2d3316-f9d0-4812-89a3-d34efcc04628" containerName="insights-gathering" containerID="cri-o://db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c" gracePeriod=30
Nov 03 19:32:12.696164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.696135    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:12.703617 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.703597    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:32:12.802469 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.802426    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-trusted-ca-bundle\") pod \"ba2d3316-f9d0-4812-89a3-d34efcc04628\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") "
Nov 03 19:32:12.802648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.802489    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-service-ca-bundle\") pod \"ba2d3316-f9d0-4812-89a3-d34efcc04628\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") "
Nov 03 19:32:12.802648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.802594    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba2d3316-f9d0-4812-89a3-d34efcc04628-kube-api-access-xng5d\" (UniqueName: \"kubernetes.io/projected/ba2d3316-f9d0-4812-89a3-d34efcc04628-kube-api-access-xng5d\") pod \"ba2d3316-f9d0-4812-89a3-d34efcc04628\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") "
Nov 03 19:32:12.802763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.802677    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba2d3316-f9d0-4812-89a3-d34efcc04628-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba2d3316-f9d0-4812-89a3-d34efcc04628-archives-path\") pod \"ba2d3316-f9d0-4812-89a3-d34efcc04628\" (UID: \"ba2d3316-f9d0-4812-89a3-d34efcc04628\") "
Nov 03 19:32:12.802817 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.802782    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-service-ca-bundle" pod "ba2d3316-f9d0-4812-89a3-d34efcc04628" (UID: "ba2d3316-f9d0-4812-89a3-d34efcc04628"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:12.802817 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.802794    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-trusted-ca-bundle" pod "ba2d3316-f9d0-4812-89a3-d34efcc04628" (UID: "ba2d3316-f9d0-4812-89a3-d34efcc04628"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:12.802984 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.802966    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-trusted-ca-bundle\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:12.803050 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.802987    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2d3316-f9d0-4812-89a3-d34efcc04628-service-ca-bundle\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:12.803050 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.802970    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2d3316-f9d0-4812-89a3-d34efcc04628-archives-path" pod "ba2d3316-f9d0-4812-89a3-d34efcc04628" (UID: "ba2d3316-f9d0-4812-89a3-d34efcc04628"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:12.804846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.804824    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2d3316-f9d0-4812-89a3-d34efcc04628-kube-api-access-xng5d" pod "ba2d3316-f9d0-4812-89a3-d34efcc04628" (UID: "ba2d3316-f9d0-4812-89a3-d34efcc04628"). InnerVolumeSpecName "kube-api-access-xng5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:12.825662 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.825626    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:12.903691 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.903654    4531 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ba2d3316-f9d0-4812-89a3-d34efcc04628-archives-path\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:12.903691 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:12.903686    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xng5d\" (UniqueName: \"kubernetes.io/projected/ba2d3316-f9d0-4812-89a3-d34efcc04628-kube-api-access-xng5d\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:13.121942 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:13.121859    4531 generic.go:356] "Generic (PLEG): container finished" podID="ba2d3316-f9d0-4812-89a3-d34efcc04628" containerID="db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c" exitCode=2
Nov 03 19:32:13.121942 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:13.121900    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9xxl7-qt48t" event={"ID":"ba2d3316-f9d0-4812-89a3-d34efcc04628","Type":"ContainerDied","Data":"db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c"}
Nov 03 19:32:13.122151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:13.121916    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9xxl7-qt48t"
Nov 03 19:32:13.122151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:13.121960    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9xxl7-qt48t" event={"ID":"ba2d3316-f9d0-4812-89a3-d34efcc04628","Type":"ContainerDied","Data":"16e2877f87a0befb90ed4eaf6833e145f834da44968f3b1c2bea606ef7a08fc5"}
Nov 03 19:32:13.122151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:13.121981    4531 scope.go:122] "RemoveContainer" containerID="db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c"
Nov 03 19:32:13.130312 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:13.130295    4531 scope.go:122] "RemoveContainer" containerID="db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c"
Nov 03 19:32:13.130630 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:32:13.130605    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c\": container with ID starting with db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c not found: ID does not exist" containerID="db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c"
Nov 03 19:32:13.130749 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:13.130637    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c"} err="failed to get container status \"db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c\": rpc error: code = NotFound desc = could not find container \"db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c\": container with ID starting with db163ad0a939f075af5e42be2ead0ddaba419e3ca20d3b4be8573c138259938c not found: ID does not exist"
Nov 03 19:32:13.139079 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:13.139058    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-9xxl7-qt48t"]
Nov 03 19:32:13.142421 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:13.142398    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-9xxl7-qt48t"]
Nov 03 19:32:13.431754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:13.431689    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:13.568303 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:13.568265    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:14.600260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:14.600227    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:14.755157 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:14.755126    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:14.855308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:14.855236    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba2d3316-f9d0-4812-89a3-d34efcc04628" path="/var/lib/kubelet/pods/ba2d3316-f9d0-4812-89a3-d34efcc04628/volumes"
Nov 03 19:32:20.572666 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:32:20.572631    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:32:27.955705 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:27.955670    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:28.084797 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:28.084767    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:28.426081 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:28.426046    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:28.563069 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:28.563037    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:32.421959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:32.421922    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:32:32.425891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:32.425868    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84bd6f669d-jvbjq"
Nov 03 19:32:34.124962 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.124914    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67d657d848-gvgc6" podUID="02690ad5-61e0-4357-aa9e-4d3530469ba1" containerName="console" containerID="cri-o://73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c" gracePeriod=15
Nov 03 19:32:34.358929 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.358901    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:32:34.370242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370207    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-serving-cert\") pod \"02690ad5-61e0-4357-aa9e-4d3530469ba1\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") "
Nov 03 19:32:34.370419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370252    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02690ad5-61e0-4357-aa9e-4d3530469ba1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/02690ad5-61e0-4357-aa9e-4d3530469ba1-tmp\") pod \"02690ad5-61e0-4357-aa9e-4d3530469ba1\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") "
Nov 03 19:32:34.370419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370298    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02690ad5-61e0-4357-aa9e-4d3530469ba1-kube-api-access-t4b6v\" (UniqueName: \"kubernetes.io/projected/02690ad5-61e0-4357-aa9e-4d3530469ba1-kube-api-access-t4b6v\") pod \"02690ad5-61e0-4357-aa9e-4d3530469ba1\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") "
Nov 03 19:32:34.370419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370328    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-config\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-config\") pod \"02690ad5-61e0-4357-aa9e-4d3530469ba1\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") "
Nov 03 19:32:34.370419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370355    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-oauth-config\") pod \"02690ad5-61e0-4357-aa9e-4d3530469ba1\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") "
Nov 03 19:32:34.370419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370389    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-service-ca\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-service-ca\") pod \"02690ad5-61e0-4357-aa9e-4d3530469ba1\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") "
Nov 03 19:32:34.370659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370470    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-trusted-ca-bundle\") pod \"02690ad5-61e0-4357-aa9e-4d3530469ba1\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") "
Nov 03 19:32:34.370659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370498    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-oauth-serving-cert\") pod \"02690ad5-61e0-4357-aa9e-4d3530469ba1\" (UID: \"02690ad5-61e0-4357-aa9e-4d3530469ba1\") "
Nov 03 19:32:34.370659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370591    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02690ad5-61e0-4357-aa9e-4d3530469ba1-tmp" pod "02690ad5-61e0-4357-aa9e-4d3530469ba1" (UID: "02690ad5-61e0-4357-aa9e-4d3530469ba1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:34.370792 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370705    4531 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02690ad5-61e0-4357-aa9e-4d3530469ba1-tmp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:34.370934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370892    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-config" pod "02690ad5-61e0-4357-aa9e-4d3530469ba1" (UID: "02690ad5-61e0-4357-aa9e-4d3530469ba1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:34.370934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370914    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-trusted-ca-bundle" pod "02690ad5-61e0-4357-aa9e-4d3530469ba1" (UID: "02690ad5-61e0-4357-aa9e-4d3530469ba1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:34.371093 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.370935    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-oauth-serving-cert" pod "02690ad5-61e0-4357-aa9e-4d3530469ba1" (UID: "02690ad5-61e0-4357-aa9e-4d3530469ba1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:34.371093 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.371005    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-service-ca" pod "02690ad5-61e0-4357-aa9e-4d3530469ba1" (UID: "02690ad5-61e0-4357-aa9e-4d3530469ba1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:34.372607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.372579    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-serving-cert" pod "02690ad5-61e0-4357-aa9e-4d3530469ba1" (UID: "02690ad5-61e0-4357-aa9e-4d3530469ba1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:32:34.372713 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.372625    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-oauth-config" pod "02690ad5-61e0-4357-aa9e-4d3530469ba1" (UID: "02690ad5-61e0-4357-aa9e-4d3530469ba1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:32:34.372713 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.372646    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02690ad5-61e0-4357-aa9e-4d3530469ba1-kube-api-access-t4b6v" pod "02690ad5-61e0-4357-aa9e-4d3530469ba1" (UID: "02690ad5-61e0-4357-aa9e-4d3530469ba1"). InnerVolumeSpecName "kube-api-access-t4b6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:34.471890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.471844    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:34.471890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.471881    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-trusted-ca-bundle\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:34.471890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.471895    4531 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-oauth-serving-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:34.472148 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.471909    4531 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-serving-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:34.472148 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.471921    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4b6v\" (UniqueName: \"kubernetes.io/projected/02690ad5-61e0-4357-aa9e-4d3530469ba1-kube-api-access-t4b6v\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:34.472148 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.471934    4531 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-config\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:34.472148 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:34.471945    4531 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/02690ad5-61e0-4357-aa9e-4d3530469ba1-console-oauth-config\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:35.205099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:35.205065    4531 generic.go:356] "Generic (PLEG): container finished" podID="02690ad5-61e0-4357-aa9e-4d3530469ba1" containerID="73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c" exitCode=0
Nov 03 19:32:35.205550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:35.205112    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d657d848-gvgc6" event={"ID":"02690ad5-61e0-4357-aa9e-4d3530469ba1","Type":"ContainerDied","Data":"73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c"}
Nov 03 19:32:35.205550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:35.205130    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d657d848-gvgc6"
Nov 03 19:32:35.205550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:35.205153    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d657d848-gvgc6" event={"ID":"02690ad5-61e0-4357-aa9e-4d3530469ba1","Type":"ContainerDied","Data":"4b7e0b8479638c9393691b348fee3375ae059d5869d1574327bf06ca42750730"}
Nov 03 19:32:35.205550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:35.205197    4531 scope.go:122] "RemoveContainer" containerID="73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c"
Nov 03 19:32:35.213556 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:35.213537    4531 scope.go:122] "RemoveContainer" containerID="73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c"
Nov 03 19:32:35.213852 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:32:35.213835    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c\": container with ID starting with 73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c not found: ID does not exist" containerID="73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c"
Nov 03 19:32:35.213900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:35.213863    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c"} err="failed to get container status \"73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c\": rpc error: code = NotFound desc = could not find container \"73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c\": container with ID starting with 73865443ad7fbb7e8c527da7c144dc0276e7be3636cf4d3e4c2ad556ea88c61c not found: ID does not exist"
Nov 03 19:32:35.218249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:35.218215    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d657d848-gvgc6"]
Nov 03 19:32:35.221706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:35.221675    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67d657d848-gvgc6"]
Nov 03 19:32:36.855366 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:36.855336    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02690ad5-61e0-4357-aa9e-4d3530469ba1" path="/var/lib/kubelet/pods/02690ad5-61e0-4357-aa9e-4d3530469ba1/volumes"
Nov 03 19:32:37.666830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:37.666723    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:43.420975 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:43.420933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:43.545274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:43.545236    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:43.802510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:43.802427    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:43.930427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:43.930388    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:44.195619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.195584    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b4b4fdf6-rzlv6"]
Nov 03 19:32:44.195909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.195895    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ba2d3316-f9d0-4812-89a3-d34efcc04628" containerName="insights-gathering"
Nov 03 19:32:44.195950 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.195910    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2d3316-f9d0-4812-89a3-d34efcc04628" containerName="insights-gathering"
Nov 03 19:32:44.195950 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.195922    4531 cpu_manager.go:414] "containerMap: removing container" podUID="02690ad5-61e0-4357-aa9e-4d3530469ba1" containerName="console"
Nov 03 19:32:44.195950 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.195927    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02690ad5-61e0-4357-aa9e-4d3530469ba1" containerName="console"
Nov 03 19:32:44.196038 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.195999    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2d3316-f9d0-4812-89a3-d34efcc04628" containerName="insights-gathering"
Nov 03 19:32:44.196038 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.196008    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="02690ad5-61e0-4357-aa9e-4d3530469ba1" containerName="console"
Nov 03 19:32:44.198831 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.198815    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.202435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.202413    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 19:32:44.202555 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.202446    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 19:32:44.202555 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.202450    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 19:32:44.205947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.205923    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-46fdn\""
Nov 03 19:32:44.206232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.206216    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 19:32:44.210475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.210453    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4b4fdf6-rzlv6"]
Nov 03 19:32:44.248316 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.248281    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/825cd9ed-888e-4788-923a-4aef03b5d537-tmp\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.248488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.248325    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-console-config\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.248488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.248352    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-service-ca\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.248488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.248413    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8gb\" (UniqueName: \"kubernetes.io/projected/825cd9ed-888e-4788-923a-4aef03b5d537-kube-api-access-6c8gb\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.248488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.248450    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/825cd9ed-888e-4788-923a-4aef03b5d537-console-oauth-config\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.248623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.248521    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-oauth-serving-cert\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.248623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.248542    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-trusted-ca-bundle\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.248623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.248572    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/825cd9ed-888e-4788-923a-4aef03b5d537-console-serving-cert\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.302225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.302190    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 19:32:44.302390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.302254    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 19:32:44.302390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.302258    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 19:32:44.302390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.302349    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 19:32:44.328201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.328149    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:44.349236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.349207    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/825cd9ed-888e-4788-923a-4aef03b5d537-console-serving-cert\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.349414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.349245    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/825cd9ed-888e-4788-923a-4aef03b5d537-tmp\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.349414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.349271    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-console-config\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.349414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.349293    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-service-ca\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.349414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.349308    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8gb\" (UniqueName: \"kubernetes.io/projected/825cd9ed-888e-4788-923a-4aef03b5d537-kube-api-access-6c8gb\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.349640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.349457    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/825cd9ed-888e-4788-923a-4aef03b5d537-console-oauth-config\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.349640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.349503    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-oauth-serving-cert\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.349640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.349531    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-trusted-ca-bundle\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.349640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.349629    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/825cd9ed-888e-4788-923a-4aef03b5d537-tmp\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.350097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.350069    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-console-config\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.350260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.350071    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-service-ca\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.350260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.350209    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-oauth-serving-cert\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.350535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.350511    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/825cd9ed-888e-4788-923a-4aef03b5d537-trusted-ca-bundle\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.352003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.351985    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/825cd9ed-888e-4788-923a-4aef03b5d537-console-oauth-config\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.352336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.352316    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/825cd9ed-888e-4788-923a-4aef03b5d537-console-serving-cert\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.356335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.356304    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8gb\" (UniqueName: \"kubernetes.io/projected/825cd9ed-888e-4788-923a-4aef03b5d537-kube-api-access-6c8gb\") pod \"console-b4b4fdf6-rzlv6\" (UID: \"825cd9ed-888e-4788-923a-4aef03b5d537\") " pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.447713 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.447633    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:44.514567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.514529    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:44.795459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.795390    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4b4fdf6-rzlv6"]
Nov 03 19:32:44.798328 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:32:44.798287    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825cd9ed_888e_4788_923a_4aef03b5d537.slice/crio-b929f84b4f64ef2626a9a6ca1241815ce56394411fc7ecfcbcb21ca128618fcd WatchSource:0}: Error finding container b929f84b4f64ef2626a9a6ca1241815ce56394411fc7ecfcbcb21ca128618fcd: Status 404 returned error can't find the container with id b929f84b4f64ef2626a9a6ca1241815ce56394411fc7ecfcbcb21ca128618fcd
Nov 03 19:32:44.874997 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:44.874961    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:45.240420 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:45.240388    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4b4fdf6-rzlv6" event={"ID":"825cd9ed-888e-4788-923a-4aef03b5d537","Type":"ContainerStarted","Data":"fa71b0966eeec7e3d42e6115417356e304c720623852355f3b3c874400c0057b"}
Nov 03 19:32:45.240586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:45.240440    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4b4fdf6-rzlv6" event={"ID":"825cd9ed-888e-4788-923a-4aef03b5d537","Type":"ContainerStarted","Data":"b929f84b4f64ef2626a9a6ca1241815ce56394411fc7ecfcbcb21ca128618fcd"}
Nov 03 19:32:45.254715 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:45.254660    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b4b4fdf6-rzlv6" podStartSLOduration=1.254644358 podStartE2EDuration="1.254644358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:45.254640134 +0000 UTC m=+78.524763430" watchObservedRunningTime="2026-11-03 19:32:45.254644358 +0000 UTC m=+78.524767646"
Nov 03 19:32:46.937845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:46.937814    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:47.059089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:47.059060    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:49.943808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.943772    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:32:49.947697 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.947676    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.950691 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.950664    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 19:32:49.950889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.950867    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 19:32:49.950980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.950905    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 19:32:49.950980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.950930    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 19:32:49.950980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.950949    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 19:32:49.950980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.950955    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5hwgk\""
Nov 03 19:32:49.951269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.951247    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 19:32:49.951414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.951394    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 19:32:49.951486 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.951402    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 19:32:49.952384 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.952365    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 19:32:49.961144 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.961121    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:32:49.995866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.995831    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-web-config\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.995866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.995868    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.996079 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.995887    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.996079 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.995947    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.996079 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.995978    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.996079 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.996034    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-config-volume\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.996079 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.996060    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-config-out\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.996258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.996079    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.996258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.996113    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.996258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.996216    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.996258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.996238    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjlzw\" (UniqueName: \"kubernetes.io/projected/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-kube-api-access-hjlzw\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.996376 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.996267    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:49.996376 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:49.996359    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.096781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.096744    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-config-out\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.096781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.096780    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.096993 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.096798    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.096993 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.096822    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.096993 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.096946    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjlzw\" (UniqueName: \"kubernetes.io/projected/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-kube-api-access-hjlzw\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.097118 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.096992    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.097118 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.097044    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.097118 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.097096    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-web-config\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.097373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.097128    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.097373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.097160    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.097373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.097211    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.097373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.097237    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.097373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.097273    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-config-volume\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.097635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.097588    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.098073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.098049    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.098290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.098056    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.099863 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.099818    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-config-out\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.100082 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.100063    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.100248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.100227    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.100299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.100233    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.100797 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.100775    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.100919 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.100882    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-config-volume\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.100977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.100952    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.101164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.101144    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.101527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.101510    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-web-config\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.104260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.104240    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjlzw\" (UniqueName: \"kubernetes.io/projected/94b1e02b-a67f-4a75-bf07-8eb0924dce9c-kube-api-access-hjlzw\") pod \"alertmanager-main-0\" (UID: \"94b1e02b-a67f-4a75-bf07-8eb0924dce9c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.275525 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.275432    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:32:50.553468 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:50.553393    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:32:50.557150 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:32:50.557121    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b1e02b_a67f_4a75_bf07_8eb0924dce9c.slice/crio-dd1ed5657c350e00375f8a19ef6557ef0e90a83933adf0bb9867751b39841699 WatchSource:0}: Error finding container dd1ed5657c350e00375f8a19ef6557ef0e90a83933adf0bb9867751b39841699: Status 404 returned error can't find the container with id dd1ed5657c350e00375f8a19ef6557ef0e90a83933adf0bb9867751b39841699
Nov 03 19:32:50.576153 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:32:50.576124    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:32:51.262759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:51.262720    4531 generic.go:356] "Generic (PLEG): container finished" podID="94b1e02b-a67f-4a75-bf07-8eb0924dce9c" containerID="5a250f7f37ae2eed2fc138a1f0162ab3671f8b4cfe9b74d8f039102249b77ae8" exitCode=0
Nov 03 19:32:51.263191 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:51.262786    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94b1e02b-a67f-4a75-bf07-8eb0924dce9c","Type":"ContainerDied","Data":"5a250f7f37ae2eed2fc138a1f0162ab3671f8b4cfe9b74d8f039102249b77ae8"}
Nov 03 19:32:51.263191 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:51.262853    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94b1e02b-a67f-4a75-bf07-8eb0924dce9c","Type":"ContainerStarted","Data":"dd1ed5657c350e00375f8a19ef6557ef0e90a83933adf0bb9867751b39841699"}
Nov 03 19:32:54.515666 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:54.515625    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:54.516146 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:54.515709    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:54.520457 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:54.520433    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:55.284846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:55.284816    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b4b4fdf6-rzlv6"
Nov 03 19:32:56.286340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:56.286305    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94b1e02b-a67f-4a75-bf07-8eb0924dce9c","Type":"ContainerStarted","Data":"3e764ffc54e30d57b46f8c2bc7d69ee83ed4614cd2d5f3dbdb203c836cb83d1e"}
Nov 03 19:32:56.286726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:56.286375    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94b1e02b-a67f-4a75-bf07-8eb0924dce9c","Type":"ContainerStarted","Data":"43bf67a10cfcd2ed3e0a84514dbd3065093c84c6ffbaa567f72473f60429bbb5"}
Nov 03 19:32:56.286726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:56.286403    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94b1e02b-a67f-4a75-bf07-8eb0924dce9c","Type":"ContainerStarted","Data":"35956eb7a76d03e9786aef6b6e6261347c8a280e6b1d5067b3e184d2e0106312"}
Nov 03 19:32:56.286726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:56.286428    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94b1e02b-a67f-4a75-bf07-8eb0924dce9c","Type":"ContainerStarted","Data":"2465da20b9e0b2c734b6de96621b179a29343b0724c851f700f8813fa700e2ff"}
Nov 03 19:32:56.286726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:56.286454    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94b1e02b-a67f-4a75-bf07-8eb0924dce9c","Type":"ContainerStarted","Data":"6ae8125a700bbf87a095ff91a684d97fa0f3bd79e0fe3ffbbe5104fbf2d18ea5"}
Nov 03 19:32:56.286726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:56.286479    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94b1e02b-a67f-4a75-bf07-8eb0924dce9c","Type":"ContainerStarted","Data":"0088282f37cae6bdc8b34f5f33fb152c93ed57b30441a7bc9f61eba55ad3eaaa"}
Nov 03 19:32:56.310134 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:56.310083    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.198263717 podStartE2EDuration="7.310069428s" totalImagesPullingTime="4.097772524s" totalInitContainerRuntime="14.033187ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:56.308473571 +0000 UTC m=+89.578596866" watchObservedRunningTime="2026-11-03 19:32:56.310069428 +0000 UTC m=+89.580192772"
Nov 03 19:32:58.442890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:58.442841    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:58.566567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:32:58.566532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:07.325260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:07.325223    4531 generic.go:356] "Generic (PLEG): container finished" podID="7a924521-7745-40c7-b46d-1a9e5b3375ea" containerID="56b10afa66abd9826bf965fcfe98860afa442188b18cedb706236b5e431cf772" exitCode=0
Nov 03 19:33:07.325656 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:07.325297    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jj9xh-5rsck" event={"ID":"7a924521-7745-40c7-b46d-1a9e5b3375ea","Type":"ContainerDied","Data":"56b10afa66abd9826bf965fcfe98860afa442188b18cedb706236b5e431cf772"}
Nov 03 19:33:08.458894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.458868    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:33:08.556005 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.555963    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-trusted-ca-bundle\") pod \"7a924521-7745-40c7-b46d-1a9e5b3375ea\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") "
Nov 03 19:33:08.556195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.556024    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-service-ca-bundle\") pod \"7a924521-7745-40c7-b46d-1a9e5b3375ea\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") "
Nov 03 19:33:08.556195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.556143    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a924521-7745-40c7-b46d-1a9e5b3375ea-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7a924521-7745-40c7-b46d-1a9e5b3375ea-archives-path\") pod \"7a924521-7745-40c7-b46d-1a9e5b3375ea\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") "
Nov 03 19:33:08.556358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.556237    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a924521-7745-40c7-b46d-1a9e5b3375ea-kube-api-access-jn62c\" (UniqueName: \"kubernetes.io/projected/7a924521-7745-40c7-b46d-1a9e5b3375ea-kube-api-access-jn62c\") pod \"7a924521-7745-40c7-b46d-1a9e5b3375ea\" (UID: \"7a924521-7745-40c7-b46d-1a9e5b3375ea\") "
Nov 03 19:33:08.556417 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.556366    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-service-ca-bundle" pod "7a924521-7745-40c7-b46d-1a9e5b3375ea" (UID: "7a924521-7745-40c7-b46d-1a9e5b3375ea"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:33:08.556468 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.556426    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-trusted-ca-bundle" pod "7a924521-7745-40c7-b46d-1a9e5b3375ea" (UID: "7a924521-7745-40c7-b46d-1a9e5b3375ea"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:33:08.556468 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.556424    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a924521-7745-40c7-b46d-1a9e5b3375ea-archives-path" pod "7a924521-7745-40c7-b46d-1a9e5b3375ea" (UID: "7a924521-7745-40c7-b46d-1a9e5b3375ea"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:33:08.558291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.558269    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a924521-7745-40c7-b46d-1a9e5b3375ea-kube-api-access-jn62c" pod "7a924521-7745-40c7-b46d-1a9e5b3375ea" (UID: "7a924521-7745-40c7-b46d-1a9e5b3375ea"). InnerVolumeSpecName "kube-api-access-jn62c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:33:08.656914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.656874    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn62c\" (UniqueName: \"kubernetes.io/projected/7a924521-7745-40c7-b46d-1a9e5b3375ea-kube-api-access-jn62c\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:08.656914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.656913    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-trusted-ca-bundle\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:08.657112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.656928    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a924521-7745-40c7-b46d-1a9e5b3375ea-service-ca-bundle\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:08.657112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:08.656941    4531 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7a924521-7745-40c7-b46d-1a9e5b3375ea-archives-path\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:09.333196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:09.333146    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jj9xh-5rsck" event={"ID":"7a924521-7745-40c7-b46d-1a9e5b3375ea","Type":"ContainerDied","Data":"67abb9ba30b5b88c5de77d07fa9e211a53bd4ce966ae475e34e26b30722dedd8"}
Nov 03 19:33:09.333196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:09.333194    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jj9xh-5rsck"
Nov 03 19:33:09.333406 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:09.333220    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67abb9ba30b5b88c5de77d07fa9e211a53bd4ce966ae475e34e26b30722dedd8"
Nov 03 19:33:14.626723 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:33:14.626682    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:20.572570 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:33:20.572529    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:33:50.572722 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:33:50.572681    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:34:14.672766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:34:14.672730    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:17.947731 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:34:17.947699    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:20.572242 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:34:20.572202    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:34:45.222948 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:34:45.222906    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:45.364540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:34:45.364485    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:50.571545 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:34:50.571506    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:35:14.707543 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:14.707459    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:20.571895 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:35:20.571860    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:35:35.833233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:35.833145    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:35.961657 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:35.961624    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:46.245769 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:46.245735    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:47.464764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.464730    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"]
Nov 03 19:35:47.467040 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.465045    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7a924521-7745-40c7-b46d-1a9e5b3375ea" containerName="insights-gathering"
Nov 03 19:35:47.467040 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.465056    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a924521-7745-40c7-b46d-1a9e5b3375ea" containerName="insights-gathering"
Nov 03 19:35:47.467040 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.465116    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a924521-7745-40c7-b46d-1a9e5b3375ea" containerName="insights-gathering"
Nov 03 19:35:47.467876 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.467861    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.470859 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.470831    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 19:35:47.471037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.470870    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 19:35:47.471037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.470888    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-79jcj\""
Nov 03 19:35:47.471037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.470896    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 19:35:47.471037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.470905    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 19:35:47.471037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.470957    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 19:35:47.471243 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.471059    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 19:35:47.482676 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.482652    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"]
Nov 03 19:35:47.552034 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.551999    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.552169 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.552034    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4m7r\" (UniqueName: \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-kube-api-access-r4m7r\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.552169 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.552072    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.552306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.552215    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.552306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.552268    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.552380 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.552356    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.552442 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.552425    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.652749 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.652719    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.652912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.652764    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.652912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.652783    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.652912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.652798    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4m7r\" (UniqueName: \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-kube-api-access-r4m7r\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.652912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.652851    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.652912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.652891    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.653193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.652927    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.653840 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.653809    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.654327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.654298    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:47.655411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.655387    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.655411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.655402    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.655685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.655656    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.655860 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.655841    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.659983 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.659959    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.660057 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.660044    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4m7r\" (UniqueName: \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-kube-api-access-r4m7r\") pod \"istiod-openshift-gateway-7876bfcb74-b5ctg\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:47.778762 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:47.778703    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:48.049651 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:48.049573    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"]
Nov 03 19:35:48.052814 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:35:48.052778    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557cb0f_d0dd_47d4_bd23_61ae82a2c72a.slice/crio-62775a543a61a195ef5fceb9bee5aa6bb73ceb6375870f05d94dce49140b7f3a WatchSource:0}: Error finding container 62775a543a61a195ef5fceb9bee5aa6bb73ceb6375870f05d94dce49140b7f3a: Status 404 returned error can't find the container with id 62775a543a61a195ef5fceb9bee5aa6bb73ceb6375870f05d94dce49140b7f3a
Nov 03 19:35:48.419301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:48.419268    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:48.887606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:48.887569    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg" event={"ID":"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a","Type":"ContainerStarted","Data":"62775a543a61a195ef5fceb9bee5aa6bb73ceb6375870f05d94dce49140b7f3a"}
Nov 03 19:35:50.572072 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:35:50.572034    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:35:52.703866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:52.699851    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:35:52.703866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:52.699992    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:35:52.904116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:52.904076    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg" event={"ID":"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a","Type":"ContainerStarted","Data":"6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43"}
Nov 03 19:35:52.904310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:52.904146    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:52.921747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:52.921696    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg" podStartSLOduration=1.278087777 podStartE2EDuration="5.921678738s" totalImagesPullingTime="4.643590961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:35:52.919633092 +0000 UTC m=+266.189756388" watchObservedRunningTime="2026-11-03 19:35:52.921678738 +0000 UTC m=+266.191802035"
Nov 03 19:35:53.274564 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:53.274531    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:53.431123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:53.431094    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:53.575389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:53.575311    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:53.702529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:53.702497    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:53.909332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:53.909303    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:35:56.181511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:35:56.181474    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:03.364791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:03.364754    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:14.761918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:14.761882    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:20.571491 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:36:20.571459    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:36:21.994544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:21.994504    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:22.123593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:22.123560    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:26.869441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:26.869412    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:36:26.870504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:26.870481    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:36:30.297391 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:30.297361    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:33.530937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:33.530900    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:33.664689 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:33.664655    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:34.728466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.728431    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:34.824917 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.824883    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"]
Nov 03 19:36:34.828620 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.828597    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:34.857908 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.857882    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"]
Nov 03 19:36:34.921700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.921670    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 19:36:34.921865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.921766    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 19:36:34.921917 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.921861    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 19:36:34.921917 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.921874    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dpmf9\""
Nov 03 19:36:34.921917 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.921909    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 19:36:34.922026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.921955    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 19:36:34.924453 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.924426    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36ce59a5-8947-48d8-af38-db7c046afae4-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:34.924614 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.924498    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36ce59a5-8947-48d8-af38-db7c046afae4-manifests\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:34.924614 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.924584    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36ce59a5-8947-48d8-af38-db7c046afae4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:34.924750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.924633    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36ce59a5-8947-48d8-af38-db7c046afae4-tmp\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:34.925008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.924987    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36ce59a5-8947-48d8-af38-db7c046afae4-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:34.925472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:34.925456    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92sw\" (UniqueName: \"kubernetes.io/projected/36ce59a5-8947-48d8-af38-db7c046afae4-kube-api-access-j92sw\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.027026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.026936    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36ce59a5-8947-48d8-af38-db7c046afae4-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.027026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.026984    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36ce59a5-8947-48d8-af38-db7c046afae4-manifests\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.027026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.027008    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36ce59a5-8947-48d8-af38-db7c046afae4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.027325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.027037    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36ce59a5-8947-48d8-af38-db7c046afae4-tmp\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.027325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.027067    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36ce59a5-8947-48d8-af38-db7c046afae4-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.027325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.027264    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92sw\" (UniqueName: \"kubernetes.io/projected/36ce59a5-8947-48d8-af38-db7c046afae4-kube-api-access-j92sw\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.027515 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.027482    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36ce59a5-8947-48d8-af38-db7c046afae4-manifests\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.029673 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.029645    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36ce59a5-8947-48d8-af38-db7c046afae4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.029673 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.029668    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36ce59a5-8947-48d8-af38-db7c046afae4-tmp\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.029851 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.029736    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36ce59a5-8947-48d8-af38-db7c046afae4-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.029851 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.029819    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36ce59a5-8947-48d8-af38-db7c046afae4-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.038846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.038823    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92sw\" (UniqueName: \"kubernetes.io/projected/36ce59a5-8947-48d8-af38-db7c046afae4-kube-api-access-j92sw\") pod \"opendatahub-operator-controller-manager-bd9746b54-4zndn\" (UID: \"36ce59a5-8947-48d8-af38-db7c046afae4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.089609 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.089574    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:35.144057 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.144022    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:35.217022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.216982    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:35.362457 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.362380    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:35.428357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.428327    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"]
Nov 03 19:36:35.432069 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:36:35.432033    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ce59a5_8947_48d8_af38_db7c046afae4.slice/crio-5b398d3d5f907bb839a8eaf95f6081980f8ebe25c20f7cf9ae703ea051ced9ae WatchSource:0}: Error finding container 5b398d3d5f907bb839a8eaf95f6081980f8ebe25c20f7cf9ae703ea051ced9ae: Status 404 returned error can't find the container with id 5b398d3d5f907bb839a8eaf95f6081980f8ebe25c20f7cf9ae703ea051ced9ae
Nov 03 19:36:35.433875 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.433858    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:36:35.496213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:35.496151    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:36.055147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:36.055108    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn" event={"ID":"36ce59a5-8947-48d8-af38-db7c046afae4","Type":"ContainerStarted","Data":"5b398d3d5f907bb839a8eaf95f6081980f8ebe25c20f7cf9ae703ea051ced9ae"}
Nov 03 19:36:38.074327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:38.074292    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:38.213227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:38.213189    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:38.357440 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:38.357412    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:38.648379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:38.648348    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:38.870280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:38.870238    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:39.069965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:39.069927    4531 generic.go:356] "Generic (PLEG): container finished" podID="36ce59a5-8947-48d8-af38-db7c046afae4" containerID="f9b1ee8c13d6c47137f8f60c6315e11151d246ff4ad0bddc3eaebac4600b8ba2" exitCode=0
Nov 03 19:36:39.070148 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:39.070010    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn" event={"ID":"36ce59a5-8947-48d8-af38-db7c046afae4","Type":"ContainerDied","Data":"f9b1ee8c13d6c47137f8f60c6315e11151d246ff4ad0bddc3eaebac4600b8ba2"}
Nov 03 19:36:40.076110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:40.076067    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn" event={"ID":"36ce59a5-8947-48d8-af38-db7c046afae4","Type":"ContainerStarted","Data":"03c57ce8effb6681249e4c52405abdf424f69b034fc4f94b617b0a0562d5b9bd"}
Nov 03 19:36:40.076572 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:40.076164    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:40.108726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:40.108678    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn" podStartSLOduration=2.288103025 podStartE2EDuration="6.108662309s" totalImagesPullingTime="3.260874919s" totalInitContainerRuntime="559.684365ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:40.106802978 +0000 UTC m=+313.376926273" watchObservedRunningTime="2026-11-03 19:36:40.108662309 +0000 UTC m=+313.378785604"
Nov 03 19:36:40.309335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:40.309295    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:40.498345 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:40.498314    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:43.313538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:43.313502    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:45.562045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:45.562010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:45.691049 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:45.691012    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:46.497577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:46.497546    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:46.624545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:46.624513    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:47.914859 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:47.914812    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:48.490124 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:48.490080    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:48.632405 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:48.632350    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:49.346751 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:49.346709    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:50.571812 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:36:50.571773    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:36:51.060734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:51.060697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:51.082638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:51.082613    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-4zndn"
Nov 03 19:36:51.476479 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:51.476448    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:51.610054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:51.610020    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:55.117969 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:55.117927    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:56.135225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:56.135192    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:57.732938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:57.732907    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:57.936363 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:57.936330    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:58.099036 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:58.098970    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:58.967706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:58.967677    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:59.110100 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:36:59.110065    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:00.561462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:00.561428    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:00.696397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:00.696365    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:01.339576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:01.339538    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:01.472623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:01.472589    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:01.704517 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:01.704475    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:01.830445 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:01.830404    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:05.512080 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:05.512048    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:05.644093 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:05.644060    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:05.906503 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:05.906473    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:06.026697 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:06.026665    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:06.299212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:06.299115    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:06.426974 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:06.426947    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:08.588469 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:08.588431    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:08.774065 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:08.774025    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:08.857210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:08.857127    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"]
Nov 03 19:37:08.868116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:08.868088    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"]
Nov 03 19:37:08.868299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:08.868205    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"
Nov 03 19:37:08.891693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:08.891664    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r6qwz\""
Nov 03 19:37:08.891693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:08.891675    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 19:37:08.892087 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:08.892038    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 19:37:08.938358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:08.938313    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:09.025878 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.025823    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvrbz\" (UniqueName: \"kubernetes.io/projected/a49e8bbf-667f-40eb-942a-7407013e89b5-kube-api-access-kvrbz\") pod \"kube-auth-proxy-f46798c78-7wgg6\" (UID: \"a49e8bbf-667f-40eb-942a-7407013e89b5\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"
Nov 03 19:37:09.026055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.025895    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49e8bbf-667f-40eb-942a-7407013e89b5-tls-certs\") pod \"kube-auth-proxy-f46798c78-7wgg6\" (UID: \"a49e8bbf-667f-40eb-942a-7407013e89b5\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"
Nov 03 19:37:09.026055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.025995    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a49e8bbf-667f-40eb-942a-7407013e89b5-tmp\") pod \"kube-auth-proxy-f46798c78-7wgg6\" (UID: \"a49e8bbf-667f-40eb-942a-7407013e89b5\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"
Nov 03 19:37:09.084351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.084308    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:09.126829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.126750    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvrbz\" (UniqueName: \"kubernetes.io/projected/a49e8bbf-667f-40eb-942a-7407013e89b5-kube-api-access-kvrbz\") pod \"kube-auth-proxy-f46798c78-7wgg6\" (UID: \"a49e8bbf-667f-40eb-942a-7407013e89b5\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"
Nov 03 19:37:09.126829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.126807    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49e8bbf-667f-40eb-942a-7407013e89b5-tls-certs\") pod \"kube-auth-proxy-f46798c78-7wgg6\" (UID: \"a49e8bbf-667f-40eb-942a-7407013e89b5\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"
Nov 03 19:37:09.126829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.126827    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a49e8bbf-667f-40eb-942a-7407013e89b5-tmp\") pod \"kube-auth-proxy-f46798c78-7wgg6\" (UID: \"a49e8bbf-667f-40eb-942a-7407013e89b5\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"
Nov 03 19:37:09.129161 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.129134    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a49e8bbf-667f-40eb-942a-7407013e89b5-tmp\") pod \"kube-auth-proxy-f46798c78-7wgg6\" (UID: \"a49e8bbf-667f-40eb-942a-7407013e89b5\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"
Nov 03 19:37:09.129320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.129303    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49e8bbf-667f-40eb-942a-7407013e89b5-tls-certs\") pod \"kube-auth-proxy-f46798c78-7wgg6\" (UID: \"a49e8bbf-667f-40eb-942a-7407013e89b5\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"
Nov 03 19:37:09.134704 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.134686    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvrbz\" (UniqueName: \"kubernetes.io/projected/a49e8bbf-667f-40eb-942a-7407013e89b5-kube-api-access-kvrbz\") pod \"kube-auth-proxy-f46798c78-7wgg6\" (UID: \"a49e8bbf-667f-40eb-942a-7407013e89b5\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"
Nov 03 19:37:09.189839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.189806    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"
Nov 03 19:37:09.223649 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.223615    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:09.459283 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.459248    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f46798c78-7wgg6"]
Nov 03 19:37:09.462277 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:37:09.462238    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49e8bbf_667f_40eb_942a_7407013e89b5.slice/crio-fc7aa3d294fd3453625ebc49e79ba22dea0c1a7982d29887245210fb8e9c8215 WatchSource:0}: Error finding container fc7aa3d294fd3453625ebc49e79ba22dea0c1a7982d29887245210fb8e9c8215: Status 404 returned error can't find the container with id fc7aa3d294fd3453625ebc49e79ba22dea0c1a7982d29887245210fb8e9c8215
Nov 03 19:37:09.926260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:09.926219    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:10.125027 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:10.124978    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:10.193820 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:10.193698    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6" event={"ID":"a49e8bbf-667f-40eb-942a-7407013e89b5","Type":"ContainerStarted","Data":"fc7aa3d294fd3453625ebc49e79ba22dea0c1a7982d29887245210fb8e9c8215"}
Nov 03 19:37:10.711547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:10.711508    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:10.911726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:10.911689    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:12.204210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:12.204112    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6" event={"ID":"a49e8bbf-667f-40eb-942a-7407013e89b5","Type":"ContainerStarted","Data":"2964780b811b7d9ca127a6d86474bc468eaf41234b96171adb76aea025c05cf4"}
Nov 03 19:37:12.223537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:12.223479    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f46798c78-7wgg6" podStartSLOduration=1.7929041190000001 podStartE2EDuration="4.223460696s" totalImagesPullingTime="2.430556577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:12.221542313 +0000 UTC m=+345.491665609" watchObservedRunningTime="2026-11-03 19:37:12.223460696 +0000 UTC m=+345.493583993"
Nov 03 19:37:12.370761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:12.370726    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:12.546279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:12.546186    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:13.741248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:13.741209    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:13.900155 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:13.900120    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:14.837307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:14.837272    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:15.444239 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.444207    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"]
Nov 03 19:37:15.447813 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.447789    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.455352 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.455321    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 19:37:15.455352 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.455327    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:37:15.455561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.455494    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 19:37:15.456145 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.455857    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-c88bt\""
Nov 03 19:37:15.456145 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.455962    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 19:37:15.456145 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.456048    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 19:37:15.457843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.457817    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"]
Nov 03 19:37:15.489838 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.489807    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12f77852-39af-4cc0-8d45-b264ca4ca198-cert\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.489996 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.489868    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12f77852-39af-4cc0-8d45-b264ca4ca198-manager-config\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.489996 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.489989    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rglqr\" (UniqueName: \"kubernetes.io/projected/12f77852-39af-4cc0-8d45-b264ca4ca198-kube-api-access-rglqr\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.490077 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.490025    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f77852-39af-4cc0-8d45-b264ca4ca198-metrics-certs\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.591327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.591288    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rglqr\" (UniqueName: \"kubernetes.io/projected/12f77852-39af-4cc0-8d45-b264ca4ca198-kube-api-access-rglqr\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.591499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.591336    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f77852-39af-4cc0-8d45-b264ca4ca198-metrics-certs\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.591499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.591391    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12f77852-39af-4cc0-8d45-b264ca4ca198-cert\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.591499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.591464    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12f77852-39af-4cc0-8d45-b264ca4ca198-manager-config\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.592228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.592204    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12f77852-39af-4cc0-8d45-b264ca4ca198-manager-config\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.593816 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.593789    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f77852-39af-4cc0-8d45-b264ca4ca198-metrics-certs\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.593938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.593918    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12f77852-39af-4cc0-8d45-b264ca4ca198-cert\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.598441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.598420    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rglqr\" (UniqueName: \"kubernetes.io/projected/12f77852-39af-4cc0-8d45-b264ca4ca198-kube-api-access-rglqr\") pod \"jobset-controller-manager-75db7bc68-7gk4z\" (UID: \"12f77852-39af-4cc0-8d45-b264ca4ca198\") " pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:15.761015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:15.760929    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:16.029931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:16.029905    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"]
Nov 03 19:37:16.032566 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:37:16.032534    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f77852_39af_4cc0_8d45_b264ca4ca198.slice/crio-f1d615ee087529288c403ce8d16a2fb8f12a87299a3c96d9721796f2ed1ba003 WatchSource:0}: Error finding container f1d615ee087529288c403ce8d16a2fb8f12a87299a3c96d9721796f2ed1ba003: Status 404 returned error can't find the container with id f1d615ee087529288c403ce8d16a2fb8f12a87299a3c96d9721796f2ed1ba003
Nov 03 19:37:16.221188 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:16.221141    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z" event={"ID":"12f77852-39af-4cc0-8d45-b264ca4ca198","Type":"ContainerStarted","Data":"f1d615ee087529288c403ce8d16a2fb8f12a87299a3c96d9721796f2ed1ba003"}
Nov 03 19:37:18.912037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:18.912007    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:19.040671 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:19.040634    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:20.576085 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:37:20.576040    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:37:21.245293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:21.245261    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z" event={"ID":"12f77852-39af-4cc0-8d45-b264ca4ca198","Type":"ContainerStarted","Data":"3fd443363b2eb9bd3e17c01e0628053b36716c73d38e43efb29656890dee4865"}
Nov 03 19:37:21.418544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:21.418508    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:21.558718 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:21.558634    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:22.249675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:22.249636    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:22.265548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:22.265498    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z" podStartSLOduration=2.116526308 podStartE2EDuration="7.265481514s" totalImagesPullingTime="5.148955206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:22.263623088 +0000 UTC m=+355.533746383" watchObservedRunningTime="2026-11-03 19:37:22.265481514 +0000 UTC m=+355.535604809"
Nov 03 19:37:24.733369 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:24.733330    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:24.871485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:24.871442    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:33.258700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:33.258663    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-75db7bc68-7gk4z"
Nov 03 19:37:34.083180 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:34.083134    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:34.211645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:34.211610    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:39.488368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:39.488329    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:39.623722 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:39.623689    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:42.099146 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:42.099106    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:42.229117 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:42.229080    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:43.681803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:43.681768    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:43.812585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:43.812551    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:44.502717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:44.502680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:44.630584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:44.630545    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:47.492727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:47.492693    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:47.619470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:47.619432    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:47.901397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:47.901366    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:48.048416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:48.048362    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:48.297337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:48.297257    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:48.421663 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:48.421635    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:50.571882 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:37:50.571845    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:37:53.950072 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:53.950035    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:54.085343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:37:54.085306    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:00.533397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:00.533362    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:00.738997 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:00.738963    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:01.886152 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:01.886118    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:02.141210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:02.141109    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:02.710899 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:02.710866    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:05.411218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:05.411187    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:05.556947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:05.556912    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:05.694847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:05.694767    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:05.898213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:05.898154    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:06.049495 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.049304    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:06.141103 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.141070    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"]
Nov 03 19:38:06.150101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.150076    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.158818 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.158789    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"]
Nov 03 19:38:06.234407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.234368    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.234581 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.234421    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.234581 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.234500    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67155b07-975c-4a86-9ef2-39587de0fc2e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.234581 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.234519    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.234687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.234608    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2vnp\" (UniqueName: \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-kube-api-access-v2vnp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.234687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.234643    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.234687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.234664    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.335528 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.335438    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vnp\" (UniqueName: \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-kube-api-access-v2vnp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.335528 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.335493    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.335528 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.335520    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.335765 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.335550    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.335765 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.335617    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.335765 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.335662    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67155b07-975c-4a86-9ef2-39587de0fc2e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.335765 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.335686    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.336507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.336476    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.338085 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.338063    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.338234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.338137    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.338234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.338161    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.338234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.338185    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67155b07-975c-4a86-9ef2-39587de0fc2e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.348461 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.348437    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.348635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.348617    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2vnp\" (UniqueName: \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-kube-api-access-v2vnp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zszqv\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.458352 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.458316    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:06.462285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.462254    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:06.613622 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.613532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:06.736291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.736265    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"]
Nov 03 19:38:06.738499 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:38:06.738469    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67155b07_975c_4a86_9ef2_39587de0fc2e.slice/crio-f895e526f161800313dc960c9695c52aecebdd03f60d8f2124c28674d1199117 WatchSource:0}: Error finding container f895e526f161800313dc960c9695c52aecebdd03f60d8f2124c28674d1199117: Status 404 returned error can't find the container with id f895e526f161800313dc960c9695c52aecebdd03f60d8f2124c28674d1199117
Nov 03 19:38:06.740602 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.740568    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:38:06.740666 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.740637    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:38:06.742325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:06.742303    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:07.433091 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:07.433059    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv" event={"ID":"67155b07-975c-4a86-9ef2-39587de0fc2e","Type":"ContainerStarted","Data":"19d92f501dbead2552582e16f4b69b6d164e3309d108220fc70d0cc1fda40fa0"}
Nov 03 19:38:07.433306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:07.433107    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv" event={"ID":"67155b07-975c-4a86-9ef2-39587de0fc2e","Type":"ContainerStarted","Data":"f895e526f161800313dc960c9695c52aecebdd03f60d8f2124c28674d1199117"}
Nov 03 19:38:07.433306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:07.433243    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:07.434746 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:07.434715    4531 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zszqv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:38:07.434848 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:07.434769    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv" podUID="67155b07-975c-4a86-9ef2-39587de0fc2e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:38:07.451711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:07.451665    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv" podStartSLOduration=1.45165193 podStartE2EDuration="1.45165193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:07.449453189 +0000 UTC m=+400.719576484" watchObservedRunningTime="2026-11-03 19:38:07.45165193 +0000 UTC m=+400.721775224"
Nov 03 19:38:08.438108 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.438080    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 19:38:08.484770 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.484736    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"]
Nov 03 19:38:08.484997 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.484953    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg" podUID="a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" containerName="discovery" containerID="cri-o://6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43" gracePeriod=30
Nov 03 19:38:08.736838 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.736813    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:38:08.754075 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.754047    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-kube-api-access-r4m7r\" (UniqueName: \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-kube-api-access-r4m7r\") pod \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") "
Nov 03 19:38:08.754219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.754089    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-ca-configmap\") pod \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") "
Nov 03 19:38:08.754219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.754136    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-local-certs\") pod \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") "
Nov 03 19:38:08.754219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.754157    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-token\" (UniqueName: \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-token\") pod \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") "
Nov 03 19:38:08.754219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.754199    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-kubeconfig\") pod \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") "
Nov 03 19:38:08.754407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.754245    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-dns-cert\") pod \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") "
Nov 03 19:38:08.754407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.754273    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-cacerts\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-cacerts\") pod \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\" (UID: \"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a\") "
Nov 03 19:38:08.755494 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.755463    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-ca-configmap" pod "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" (UID: "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:38:08.755795 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.755769    4531 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-ca-configmap\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:08.757050 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.757021    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-token" pod "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" (UID: "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:38:08.757491 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.757463    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-kubeconfig" pod "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" (UID: "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:38:08.757597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.757514    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-cacerts" pod "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" (UID: "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:38:08.757860 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.757831    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-local-certs" pod "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" (UID: "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:38:08.757860 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.757842    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-dns-cert" pod "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" (UID: "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:38:08.758016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.757997    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-kube-api-access-r4m7r" pod "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" (UID: "a557cb0f-d0dd-47d4-bd23-61ae82a2c72a"). InnerVolumeSpecName "kube-api-access-r4m7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:38:08.856834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.856798    4531 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-csr-dns-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:08.856834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.856832    4531 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-cacerts\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:08.857029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.856846    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4m7r\" (UniqueName: \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-kube-api-access-r4m7r\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:08.857029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.856863    4531 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-local-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:08.857029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.856877    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:08.857029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:08.856892    4531 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a-istio-kubeconfig\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:09.443048 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:09.443012    4531 generic.go:356] "Generic (PLEG): container finished" podID="a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" containerID="6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43" exitCode=0
Nov 03 19:38:09.443499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:09.443067    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"
Nov 03 19:38:09.443499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:09.443098    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg" event={"ID":"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a","Type":"ContainerDied","Data":"6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43"}
Nov 03 19:38:09.443499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:09.443155    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg" event={"ID":"a557cb0f-d0dd-47d4-bd23-61ae82a2c72a","Type":"ContainerDied","Data":"62775a543a61a195ef5fceb9bee5aa6bb73ceb6375870f05d94dce49140b7f3a"}
Nov 03 19:38:09.443499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:09.443192    4531 scope.go:122] "RemoveContainer" containerID="6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43"
Nov 03 19:38:09.451332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:09.451311    4531 scope.go:122] "RemoveContainer" containerID="6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43"
Nov 03 19:38:09.451590 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:38:09.451570    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43\": container with ID starting with 6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43 not found: ID does not exist" containerID="6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43"
Nov 03 19:38:09.451658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:09.451601    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43"} err="failed to get container status \"6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43\": rpc error: code = NotFound desc = could not find container \"6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43\": container with ID starting with 6e67f4d4cbe776b1a77d7f3f8a1f44be614bfb1c73fe551b26e9b838877cba43 not found: ID does not exist"
Nov 03 19:38:09.463464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:09.463414    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"]
Nov 03 19:38:09.470030 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:09.470007    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b5ctg"]
Nov 03 19:38:10.856892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:10.856859    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" path="/var/lib/kubelet/pods/a557cb0f-d0dd-47d4-bd23-61ae82a2c72a/volumes"
Nov 03 19:38:14.844877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:14.844837    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:20.572092 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:38:20.572053    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:38:22.912329 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:22.912289    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:23.049647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:23.049615    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:23.186225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:23.186106    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:26.678192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:26.678144    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:26.821302 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:26.821260    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:28.078522 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:28.078493    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:28.231494 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:28.231457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:50.572037 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:38:50.571999    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:38:59.012498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:59.012459    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:59.152545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:38:59.152503    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:00.326375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:00.326340    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:00.470959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:00.470927    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:00.610017 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:00.609930    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:01.168362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:01.168331    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:10.807468 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:10.807436    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:10.942291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:10.942255    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:14.905810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:14.905778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:18.945256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:18.945224    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:19.320617 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:19.320522    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:19.457855 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:19.457826    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:20.571771 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:39:20.571737    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:39:32.646604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:32.646569    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:32.771044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:32.771006    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:47.457635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:47.457587    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:47.597794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:39:47.597763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:50.571923 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:39:50.571889    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:40:06.370707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.370675    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"]
Nov 03 19:40:06.371242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.371028    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" containerName="discovery"
Nov 03 19:40:06.371242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.371039    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" containerName="discovery"
Nov 03 19:40:06.371242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.371095    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a557cb0f-d0dd-47d4-bd23-61ae82a2c72a" containerName="discovery"
Nov 03 19:40:06.373998 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.373971    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.380385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.380315    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 19:40:06.380618 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.380391    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 19:40:06.380717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.380645    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ctjfc\""
Nov 03 19:40:06.380816 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.380773    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 19:40:06.381403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.381379    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"]
Nov 03 19:40:06.453608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.453576    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:06.459451 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.459427    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.459524 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.459463    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c731a1ef-3377-4475-82cb-213ba543c002-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.459524 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.459494    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.459601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.459553    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncpqj\" (UniqueName: \"kubernetes.io/projected/c731a1ef-3377-4475-82cb-213ba543c002-kube-api-access-ncpqj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.459693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.459677    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.459735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.459721    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.560958 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.560918    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.561124 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.560973    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.561124 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.561095    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.561416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.561150    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c731a1ef-3377-4475-82cb-213ba543c002-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.561416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.561226    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.561416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.561252    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncpqj\" (UniqueName: \"kubernetes.io/projected/c731a1ef-3377-4475-82cb-213ba543c002-kube-api-access-ncpqj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.561416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.561405    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.561640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.561437    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.561640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.561549    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.563305 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.563283    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c731a1ef-3377-4475-82cb-213ba543c002-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.563550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.563533    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c731a1ef-3377-4475-82cb-213ba543c002-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.567761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.567740    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncpqj\" (UniqueName: \"kubernetes.io/projected/c731a1ef-3377-4475-82cb-213ba543c002-kube-api-access-ncpqj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mw5r6\" (UID: \"c731a1ef-3377-4475-82cb-213ba543c002\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.687974 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.687924    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:06.960048 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:06.960019    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"]
Nov 03 19:40:06.963134 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:40:06.963102    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc731a1ef_3377_4475_82cb_213ba543c002.slice/crio-c517f02a4a048ea9f0c760f3a6cd4e941168f89eb535978b1fd37b81b5f651a9 WatchSource:0}: Error finding container c517f02a4a048ea9f0c760f3a6cd4e941168f89eb535978b1fd37b81b5f651a9: Status 404 returned error can't find the container with id c517f02a4a048ea9f0c760f3a6cd4e941168f89eb535978b1fd37b81b5f651a9
Nov 03 19:40:07.918754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:07.918718    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6" event={"ID":"c731a1ef-3377-4475-82cb-213ba543c002","Type":"ContainerStarted","Data":"c517f02a4a048ea9f0c760f3a6cd4e941168f89eb535978b1fd37b81b5f651a9"}
Nov 03 19:40:08.678329 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:08.678288    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:08.924278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:08.924241    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6" event={"ID":"c731a1ef-3377-4475-82cb-213ba543c002","Type":"ContainerStarted","Data":"9b79afbbb3947cbc578c8124fe8ba908f90cb006fe4ad0b87188f8ee82b78450"}
Nov 03 19:40:08.939811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:08.939719    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6" podStartSLOduration=1.578894241 podStartE2EDuration="2.93970457s" totalImagesPullingTime="1.360810329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:08.937863733 +0000 UTC m=+522.207987029" watchObservedRunningTime="2026-11-03 19:40:08.93970457 +0000 UTC m=+522.209827844"
Nov 03 19:40:09.689032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:09.689004    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:11.173164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.173130    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"]
Nov 03 19:40:11.176784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.176764    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.182976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.182954    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 19:40:11.184671 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.184648    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"]
Nov 03 19:40:11.276714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.276687    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:11.303003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.302970    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gpm7\" (UniqueName: \"kubernetes.io/projected/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-kube-api-access-2gpm7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.303159 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.303020    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.303159 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.303117    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.303413 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.303241    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.303413 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.303270    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.303486 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.303408    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.404091 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.404059    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.404275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.404102    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gpm7\" (UniqueName: \"kubernetes.io/projected/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-kube-api-access-2gpm7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.404275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.404124    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.404275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.404144    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.404275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.404200    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.404275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.404224    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.404623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.404601    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.404688 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.404623    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.404732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.404712    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.406390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.406371    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.406569 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.406553    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.410857 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.410834    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gpm7\" (UniqueName: \"kubernetes.io/projected/6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a-kube-api-access-2gpm7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hwtt\" (UID: \"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.493050 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.492949    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:11.760526 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.760473    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"]
Nov 03 19:40:11.762731 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:40:11.762701    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4e2fd3_3106_4646_bb92_8f8cb9cb8c2a.slice/crio-e994e90dafb00a184102ed1c0ac7bf3f23c2ce05a298fd9aa4981a4dfc292717 WatchSource:0}: Error finding container e994e90dafb00a184102ed1c0ac7bf3f23c2ce05a298fd9aa4981a4dfc292717: Status 404 returned error can't find the container with id e994e90dafb00a184102ed1c0ac7bf3f23c2ce05a298fd9aa4981a4dfc292717
Nov 03 19:40:11.937224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:11.937184    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt" event={"ID":"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a","Type":"ContainerStarted","Data":"e994e90dafb00a184102ed1c0ac7bf3f23c2ce05a298fd9aa4981a4dfc292717"}
Nov 03 19:40:12.943054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:12.943020    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt" event={"ID":"6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a","Type":"ContainerStarted","Data":"6834d6c4c5087eba19a16c646f44cb42643d215e8862840b5d9c8f7d5d88c624"}
Nov 03 19:40:12.957731 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:12.957655    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt" podStartSLOduration=1.6262327079999999 podStartE2EDuration="1.957638087s" totalImagesPullingTime="331.405379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:12.957456498 +0000 UTC m=+526.227579794" watchObservedRunningTime="2026-11-03 19:40:12.957638087 +0000 UTC m=+526.227761386"
Nov 03 19:40:13.379369 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.379336    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:13.493991 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.493956    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:13.812791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.812717    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-nsvcq"]
Nov 03 19:40:13.816426 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.816405    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:13.823067 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.823046    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-nsvcq"]
Nov 03 19:40:13.823927 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.823906    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 19:40:13.908661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.908632    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:13.925382 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.925357    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:13.925540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.925396    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-home\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:13.925540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.925506    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:13.925773 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.925750    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:13.925834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.925817    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2p8\" (UniqueName: \"kubernetes.io/projected/bd23398f-be9f-410a-aa58-7f2af78f8169-kube-api-access-cv2p8\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:13.925882 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:13.925866    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd23398f-be9f-410a-aa58-7f2af78f8169-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.027043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.027002    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.027043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.027041    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2p8\" (UniqueName: \"kubernetes.io/projected/bd23398f-be9f-410a-aa58-7f2af78f8169-kube-api-access-cv2p8\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.027812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.027064    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd23398f-be9f-410a-aa58-7f2af78f8169-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.027812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.027221    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.027812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.027318    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-home\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.027812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.027360    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.027812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.027622    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.027812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.027694    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-home\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.027812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.027707    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.029375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.029356    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd23398f-be9f-410a-aa58-7f2af78f8169-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.029545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.029528    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd23398f-be9f-410a-aa58-7f2af78f8169-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.035037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.035009    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2p8\" (UniqueName: \"kubernetes.io/projected/bd23398f-be9f-410a-aa58-7f2af78f8169-kube-api-access-cv2p8\") pod \"echo-simulator-kserve-64b6f8f96c-nsvcq\" (UID: \"bd23398f-be9f-410a-aa58-7f2af78f8169\") " pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.133147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.133113    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:14.407513 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.407490    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-nsvcq"]
Nov 03 19:40:14.409814 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:40:14.409783    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd23398f_be9f_410a_aa58_7f2af78f8169.slice/crio-82e5e58a4355b2e192354bed83eed8cf4a403745c1c93d9180f438d2856ce532 WatchSource:0}: Error finding container 82e5e58a4355b2e192354bed83eed8cf4a403745c1c93d9180f438d2856ce532: Status 404 returned error can't find the container with id 82e5e58a4355b2e192354bed83eed8cf4a403745c1c93d9180f438d2856ce532
Nov 03 19:40:14.952230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:14.952196    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq" event={"ID":"bd23398f-be9f-410a-aa58-7f2af78f8169","Type":"ContainerStarted","Data":"82e5e58a4355b2e192354bed83eed8cf4a403745c1c93d9180f438d2856ce532"}
Nov 03 19:40:15.080751 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:15.080702    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:16.035978 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:16.035937    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:16.688732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:16.688692    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:16.701449 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:16.701425    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:16.962854 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:16.962765    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq" event={"ID":"bd23398f-be9f-410a-aa58-7f2af78f8169","Type":"ContainerStarted","Data":"89d051932c36a030be1bd106f85596eedde3203ad950987981a6a7a64a83e2b5"}
Nov 03 19:40:16.976606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:16.976580    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:40:16.987831 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:16.987780    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq" podStartSLOduration=2.290414347 podStartE2EDuration="3.987765543s" totalImagesPullingTime="1.697351196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:16.986984439 +0000 UTC m=+530.257107736" watchObservedRunningTime="2026-11-03 19:40:16.987765543 +0000 UTC m=+530.257888872"
Nov 03 19:40:17.008094 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.008062    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"]
Nov 03 19:40:17.011799 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.011777    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.022374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.022340    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 19:40:17.022512 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.022433    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"]
Nov 03 19:40:17.119055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.118979    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:17.134107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.134079    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:17.155520 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.155489    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.155647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.155532    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.155647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.155561    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.155742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.155645    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8v9g\" (UniqueName: \"kubernetes.io/projected/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-kube-api-access-c8v9g\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.155786 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.155751    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.155871 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.155808    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.256682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.256603    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.256682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.256646    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.256682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.256667    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.256937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.256757    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.256937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.256787    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8v9g\" (UniqueName: \"kubernetes.io/projected/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-kube-api-access-c8v9g\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.256937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.256868    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.257103 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.257069    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.257103 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.257088    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.257254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.257238    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.259056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.259036    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.259372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.259349    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.263669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.263648    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8v9g\" (UniqueName: \"kubernetes.io/projected/3b05d47f-88b5-45ce-9af3-cbde59ec2d69-kube-api-access-c8v9g\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8\" (UID: \"3b05d47f-88b5-45ce-9af3-cbde59ec2d69\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.332879 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.332847    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:17.601546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.601522    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"]
Nov 03 19:40:17.603932 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:40:17.603906    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b05d47f_88b5_45ce_9af3_cbde59ec2d69.slice/crio-187967f220c05277d88a1d86ba4b597e6fe471827605e965ac524bcb882794df WatchSource:0}: Error finding container 187967f220c05277d88a1d86ba4b597e6fe471827605e965ac524bcb882794df: Status 404 returned error can't find the container with id 187967f220c05277d88a1d86ba4b597e6fe471827605e965ac524bcb882794df
Nov 03 19:40:17.968736 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:17.968705    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" event={"ID":"3b05d47f-88b5-45ce-9af3-cbde59ec2d69","Type":"ContainerStarted","Data":"187967f220c05277d88a1d86ba4b597e6fe471827605e965ac524bcb882794df"}
Nov 03 19:40:18.974238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:18.974204    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" event={"ID":"3b05d47f-88b5-45ce-9af3-cbde59ec2d69","Type":"ContainerStarted","Data":"7afd145cb9aadf567255335a8cb5e0e2e01feb92f10dea608bcf9a166b17a762"}
Nov 03 19:40:18.988920 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:18.988875    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podStartSLOduration=2.642074661 podStartE2EDuration="2.988860971s" totalImagesPullingTime="346.78631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:18.988002848 +0000 UTC m=+532.258126141" watchObservedRunningTime="2026-11-03 19:40:18.988860971 +0000 UTC m=+532.258984266"
Nov 03 19:40:19.333163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:19.333080    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:20.150520 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:20.150489    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:20.571967 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:40:20.571891    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:40:21.494004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:21.493965    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:21.506368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:21.506344    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:21.997195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:21.997145    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:40:24.133649 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:24.133610    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:24.146005 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:24.145968    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:25.009085 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:25.009058    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-nsvcq"
Nov 03 19:40:27.333860 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:27.333826    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:27.346122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:27.346100    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:28.021478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:28.021450    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:40:36.673222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.673167    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-tk99k"]
Nov 03 19:40:36.679870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.679854    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:40:36.684863 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.684836    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 19:40:36.685049 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.684866    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-9rvrt\""
Nov 03 19:40:36.685049 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.684930    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 19:40:36.685238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.685089    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 19:40:36.685451 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.685431    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 19:40:36.687017 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.686992    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-tk99k"]
Nov 03 19:40:36.716274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.716244    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:36.836843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.836801    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96cml\" (UniqueName: \"kubernetes.io/projected/ae7a981e-d31b-4410-9502-5d7a90f381e9-kube-api-access-96cml\") pod \"ai-gateway-controller-55cd769764-tk99k\" (UID: \"ae7a981e-d31b-4410-9502-5d7a90f381e9\") " pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:40:36.836843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.836848    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7a981e-d31b-4410-9502-5d7a90f381e9-metrics-certs\") pod \"ai-gateway-controller-55cd769764-tk99k\" (UID: \"ae7a981e-d31b-4410-9502-5d7a90f381e9\") " pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:40:36.845607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.845570    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:36.937923 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.937841    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7a981e-d31b-4410-9502-5d7a90f381e9-metrics-certs\") pod \"ai-gateway-controller-55cd769764-tk99k\" (UID: \"ae7a981e-d31b-4410-9502-5d7a90f381e9\") " pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:40:36.937923 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.937924    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96cml\" (UniqueName: \"kubernetes.io/projected/ae7a981e-d31b-4410-9502-5d7a90f381e9-kube-api-access-96cml\") pod \"ai-gateway-controller-55cd769764-tk99k\" (UID: \"ae7a981e-d31b-4410-9502-5d7a90f381e9\") " pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:40:36.940354 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.940324    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7a981e-d31b-4410-9502-5d7a90f381e9-metrics-certs\") pod \"ai-gateway-controller-55cd769764-tk99k\" (UID: \"ae7a981e-d31b-4410-9502-5d7a90f381e9\") " pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:40:36.944915 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.944891    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96cml\" (UniqueName: \"kubernetes.io/projected/ae7a981e-d31b-4410-9502-5d7a90f381e9-kube-api-access-96cml\") pod \"ai-gateway-controller-55cd769764-tk99k\" (UID: \"ae7a981e-d31b-4410-9502-5d7a90f381e9\") " pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:40:36.994664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:36.994635    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:40:37.271246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:37.271222    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-tk99k"]
Nov 03 19:40:37.273789 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:40:37.273757    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7a981e_d31b_4410_9502_5d7a90f381e9.slice/crio-d0e9b4a66498ca1b1e682357e0ced7d9ac90bf7121db40ad0dbb6577f25bec18 WatchSource:0}: Error finding container d0e9b4a66498ca1b1e682357e0ced7d9ac90bf7121db40ad0dbb6577f25bec18: Status 404 returned error can't find the container with id d0e9b4a66498ca1b1e682357e0ced7d9ac90bf7121db40ad0dbb6577f25bec18
Nov 03 19:40:38.058609 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.058578    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k" event={"ID":"ae7a981e-d31b-4410-9502-5d7a90f381e9","Type":"ContainerStarted","Data":"d0e9b4a66498ca1b1e682357e0ced7d9ac90bf7121db40ad0dbb6577f25bec18"}
Nov 03 19:40:38.729609 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.729570    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"]
Nov 03 19:40:38.736950 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.736924    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.743099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.743042    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"]
Nov 03 19:40:38.744552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.744232    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7lc4h\""
Nov 03 19:40:38.756505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.756475    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.756643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.756528    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.756643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.756630    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.756760 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.756674    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.756760 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.756703    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf82t\" (UniqueName: \"kubernetes.io/projected/bd352072-e76d-4d10-93a1-a71b2e0495ab-kube-api-access-gf82t\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.756760 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.756737    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"]
Nov 03 19:40:38.756911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.756774    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.756911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.756817    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.756911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.756819    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.756911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.756860    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd352072-e76d-4d10-93a1-a71b2e0495ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.756911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.756908    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.766973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.766947    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"]
Nov 03 19:40:38.857950 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.857915    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.858132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.857976    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.858132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.857994    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.858132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858031    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.858132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858060    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.858132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858090    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.858132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858107    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.858399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858138    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.858399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858154    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf82t\" (UniqueName: \"kubernetes.io/projected/bd352072-e76d-4d10-93a1-a71b2e0495ab-kube-api-access-gf82t\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.858399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858222    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.858399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858258    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.858399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858279    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.858399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858320    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.858399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858341    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.858399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858361    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.858657 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858440    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd352072-e76d-4d10-93a1-a71b2e0495ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.858657 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858458    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.858657 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.858473    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw6qg\" (UniqueName: \"kubernetes.io/projected/c6beddf2-5be3-42ae-a6c7-6256834bd41f-kube-api-access-hw6qg\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.859767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.859436    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.859767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.859471    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.859767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.859524    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.860029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.859949    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.860029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.859963    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd352072-e76d-4d10-93a1-a71b2e0495ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.861751 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.861727    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.867541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.867516    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.868109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.868093    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf82t\" (UniqueName: \"kubernetes.io/projected/bd352072-e76d-4d10-93a1-a71b2e0495ab-kube-api-access-gf82t\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.870652 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.870630    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd352072-e76d-4d10-93a1-a71b2e0495ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7n48w\" (UID: \"bd352072-e76d-4d10-93a1-a71b2e0495ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:38.959046 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959009    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.959241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959051    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.959241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959090    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.959241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959113    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6qg\" (UniqueName: \"kubernetes.io/projected/c6beddf2-5be3-42ae-a6c7-6256834bd41f-kube-api-access-hw6qg\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.959241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959200    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.959241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959230    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.959515 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959269    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.959515 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959337    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.959515 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959383    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.959515 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959486    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.959515 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959499    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.959878 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.959852    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.960048 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.960028    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.960123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.960084    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.962231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.962209    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.962451 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.962432    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.967768 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.967434    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6beddf2-5be3-42ae-a6c7-6256834bd41f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:38.967768 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:38.967729    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6qg\" (UniqueName: \"kubernetes.io/projected/c6beddf2-5be3-42ae-a6c7-6256834bd41f-kube-api-access-hw6qg\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-t5g9v\" (UID: \"c6beddf2-5be3-42ae-a6c7-6256834bd41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:39.062203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:39.062107    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:39.078944 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:39.078919    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:39.346480 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:39.346447    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"]
Nov 03 19:40:39.348385 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:40:39.348357    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd352072_e76d_4d10_93a1_a71b2e0495ab.slice/crio-4ea85be158b3eac6260dd0ab7e5f13b9f1812720cc1907bd207e932fad45e5e1 WatchSource:0}: Error finding container 4ea85be158b3eac6260dd0ab7e5f13b9f1812720cc1907bd207e932fad45e5e1: Status 404 returned error can't find the container with id 4ea85be158b3eac6260dd0ab7e5f13b9f1812720cc1907bd207e932fad45e5e1
Nov 03 19:40:39.371555 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:39.371530    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"]
Nov 03 19:40:39.373071 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:40:39.373039    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6beddf2_5be3_42ae_a6c7_6256834bd41f.slice/crio-7499a704e5722254bf8690f346bbc2c813e0604900437482b83e8cd0b0abe048 WatchSource:0}: Error finding container 7499a704e5722254bf8690f346bbc2c813e0604900437482b83e8cd0b0abe048: Status 404 returned error can't find the container with id 7499a704e5722254bf8690f346bbc2c813e0604900437482b83e8cd0b0abe048
Nov 03 19:40:40.069689 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:40.069596    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v" event={"ID":"c6beddf2-5be3-42ae-a6c7-6256834bd41f","Type":"ContainerStarted","Data":"7499a704e5722254bf8690f346bbc2c813e0604900437482b83e8cd0b0abe048"}
Nov 03 19:40:40.071314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:40.071279    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k" event={"ID":"ae7a981e-d31b-4410-9502-5d7a90f381e9","Type":"ContainerStarted","Data":"de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616"}
Nov 03 19:40:40.071466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:40.071445    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:40:40.072847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:40.072826    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" event={"ID":"bd352072-e76d-4d10-93a1-a71b2e0495ab","Type":"ContainerStarted","Data":"4ea85be158b3eac6260dd0ab7e5f13b9f1812720cc1907bd207e932fad45e5e1"}
Nov 03 19:40:40.086516 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:40.086437    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k" podStartSLOduration=1.567737772 podStartE2EDuration="4.086371183s" totalImagesPullingTime="2.518633411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:40.085321982 +0000 UTC m=+553.355445268" watchObservedRunningTime="2026-11-03 19:40:40.086371183 +0000 UTC m=+553.356494500"
Nov 03 19:40:40.629426 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:40.629367    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:40.766140 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:40.766106    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:42.386168 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:42.386135    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:42.386463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:42.386233    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:42.386463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:42.386266    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:43.091737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:43.091693    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v" event={"ID":"c6beddf2-5be3-42ae-a6c7-6256834bd41f","Type":"ContainerStarted","Data":"4b9d70c07263f7caf84b39f4277ffb9de655f85cf8cbe6fb085557c995539b4e"}
Nov 03 19:40:43.109518 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:43.109460    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v" podStartSLOduration=2.099359198 podStartE2EDuration="5.109442423s" totalImagesPullingTime="3.010083225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:43.107768968 +0000 UTC m=+556.377892263" watchObservedRunningTime="2026-11-03 19:40:43.109442423 +0000 UTC m=+556.379565718"
Nov 03 19:40:43.565772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:43.565736    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:43.566043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:43.565810    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:43.566043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:43.565842    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:44.081048 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:44.081012    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:44.081450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:44.081374    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 19:40:44.081606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:44.081584    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v" podUID="c6beddf2-5be3-42ae-a6c7-6256834bd41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 19:40:44.099774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:44.099741    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" event={"ID":"bd352072-e76d-4d10-93a1-a71b2e0495ab","Type":"ContainerStarted","Data":"146a9b2613960bc08b040f3799d9febd705f66108ddfa453ed0fed2beae339de"}
Nov 03 19:40:44.118226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:44.118166    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" podStartSLOduration=1.90347374 podStartE2EDuration="6.118150513s" totalImagesPullingTime="4.214676773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:44.115611074 +0000 UTC m=+557.385734369" watchObservedRunningTime="2026-11-03 19:40:44.118150513 +0000 UTC m=+557.388273856"
Nov 03 19:40:45.068058 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:45.068020    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:45.068654 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:45.068304    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7n48w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:40:45.068834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:45.068807    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" podUID="bd352072-e76d-4d10-93a1-a71b2e0495ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:40:45.079771 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:45.079738    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 19:40:45.080033 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:45.080005    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v" podUID="c6beddf2-5be3-42ae-a6c7-6256834bd41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 19:40:46.064197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:46.062612    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7n48w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:40:46.064197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:46.062676    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" podUID="bd352072-e76d-4d10-93a1-a71b2e0495ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:40:46.085974 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:46.085566    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 19:40:46.085974 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:46.085625    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v" podUID="c6beddf2-5be3-42ae-a6c7-6256834bd41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 19:40:47.069825 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:47.069785    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7n48w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:40:47.088202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:47.085254    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" podUID="bd352072-e76d-4d10-93a1-a71b2e0495ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:40:47.088202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:47.079347    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 19:40:47.088202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:47.085712    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v" podUID="c6beddf2-5be3-42ae-a6c7-6256834bd41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 19:40:48.063196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:48.062580    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7n48w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:40:48.063196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:48.062643    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" podUID="bd352072-e76d-4d10-93a1-a71b2e0495ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:40:48.079554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:48.079517    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 19:40:48.079833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:48.079806    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v" podUID="c6beddf2-5be3-42ae-a6c7-6256834bd41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 19:40:49.062655 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:49.062616    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7n48w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:40:49.063282 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:49.063248    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" podUID="bd352072-e76d-4d10-93a1-a71b2e0495ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:40:49.079604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:49.079578    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 19:40:49.079747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:49.079620    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v" podUID="c6beddf2-5be3-42ae-a6c7-6256834bd41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 19:40:50.062462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:50.062423    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7n48w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:40:50.062727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:50.062492    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" podUID="bd352072-e76d-4d10-93a1-a71b2e0495ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:40:50.575311 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:40:50.575256    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:40:50.766619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:50.766584    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:50.767027 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:50.767003    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:50.768033 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:50.768009    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v"
Nov 03 19:40:51.062909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.062872    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7n48w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 19:40:51.063317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.062931    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" podUID="bd352072-e76d-4d10-93a1-a71b2e0495ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 19:40:51.083596 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.083570    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:40:51.336109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.336025    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"]
Nov 03 19:40:51.339643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.339619    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"
Nov 03 19:40:51.349956 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.349878    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"]
Nov 03 19:40:51.424310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.424265    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60aac6d-9415-41b2-8883-9eb131ded1cb-metrics-certs\") pod \"ai-gateway-controller-9bf4fd58b-7rs7s\" (UID: \"c60aac6d-9415-41b2-8883-9eb131ded1cb\") " pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"
Nov 03 19:40:51.424500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.424340    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pm48\" (UniqueName: \"kubernetes.io/projected/c60aac6d-9415-41b2-8883-9eb131ded1cb-kube-api-access-2pm48\") pod \"ai-gateway-controller-9bf4fd58b-7rs7s\" (UID: \"c60aac6d-9415-41b2-8883-9eb131ded1cb\") " pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"
Nov 03 19:40:51.525136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.525093    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pm48\" (UniqueName: \"kubernetes.io/projected/c60aac6d-9415-41b2-8883-9eb131ded1cb-kube-api-access-2pm48\") pod \"ai-gateway-controller-9bf4fd58b-7rs7s\" (UID: \"c60aac6d-9415-41b2-8883-9eb131ded1cb\") " pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"
Nov 03 19:40:51.525358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.525168    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60aac6d-9415-41b2-8883-9eb131ded1cb-metrics-certs\") pod \"ai-gateway-controller-9bf4fd58b-7rs7s\" (UID: \"c60aac6d-9415-41b2-8883-9eb131ded1cb\") " pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"
Nov 03 19:40:51.527714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.527684    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c60aac6d-9415-41b2-8883-9eb131ded1cb-metrics-certs\") pod \"ai-gateway-controller-9bf4fd58b-7rs7s\" (UID: \"c60aac6d-9415-41b2-8883-9eb131ded1cb\") " pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"
Nov 03 19:40:51.532432 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.532407    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pm48\" (UniqueName: \"kubernetes.io/projected/c60aac6d-9415-41b2-8883-9eb131ded1cb-kube-api-access-2pm48\") pod \"ai-gateway-controller-9bf4fd58b-7rs7s\" (UID: \"c60aac6d-9415-41b2-8883-9eb131ded1cb\") " pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"
Nov 03 19:40:51.654706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.654666    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"
Nov 03 19:40:51.935182 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:51.935083    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"]
Nov 03 19:40:51.937692 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:40:51.937664    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60aac6d_9415_41b2_8883_9eb131ded1cb.slice/crio-479952ec86ac3d443b4ba949b41877428bcf5f4c7d8113cfb76d213a8bbbc361 WatchSource:0}: Error finding container 479952ec86ac3d443b4ba949b41877428bcf5f4c7d8113cfb76d213a8bbbc361: Status 404 returned error can't find the container with id 479952ec86ac3d443b4ba949b41877428bcf5f4c7d8113cfb76d213a8bbbc361
Nov 03 19:40:52.066527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:52.066499    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:52.066891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:52.066874    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:52.067546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:52.067527    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w"
Nov 03 19:40:52.139362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:52.139327    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s" event={"ID":"c60aac6d-9415-41b2-8883-9eb131ded1cb","Type":"ContainerStarted","Data":"479952ec86ac3d443b4ba949b41877428bcf5f4c7d8113cfb76d213a8bbbc361"}
Nov 03 19:40:53.144386 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.144347    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s" event={"ID":"c60aac6d-9415-41b2-8883-9eb131ded1cb","Type":"ContainerStarted","Data":"323218affa99445f10dca08608d3eedded0a9426b4f91f9c6dfc881b9bcda2e9"}
Nov 03 19:40:53.144884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.144557    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"
Nov 03 19:40:53.158874 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.158829    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s" podStartSLOduration=1.7192180879999999 podStartE2EDuration="2.158816168s" totalImagesPullingTime="439.59808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:53.156784431 +0000 UTC m=+566.426907722" watchObservedRunningTime="2026-11-03 19:40:53.158816168 +0000 UTC m=+566.428939509"
Nov 03 19:40:53.767725 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.767690    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"]
Nov 03 19:40:53.771398 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.771369    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:53.780912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.780884    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"]
Nov 03 19:40:53.947361 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.947331    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tc4\" (UniqueName: \"kubernetes.io/projected/e5b033e2-ebcc-4112-99d6-d9398e5ac835-kube-api-access-w6tc4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:53.947522 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.947369    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:53.947522 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.947399    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:53.947588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.947519    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:53.947588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.947546    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:53.947588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.947568    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:53.947692 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.947672    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:53.947726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.947703    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:53.947758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:53.947730    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.048862 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.048778    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tc4\" (UniqueName: \"kubernetes.io/projected/e5b033e2-ebcc-4112-99d6-d9398e5ac835-kube-api-access-w6tc4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.048862 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.048818    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.048862 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.048844    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.049147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.048873    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.049147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.048892    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.049147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.048915    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.049147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.049081    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.049147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.049124    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.049396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.049212    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.049396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.049240    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.049474 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.049412    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.049474 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.049438    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.049568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.049549    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.049848 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.049830    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.051450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.051429    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.051641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.051625    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.055564 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.055536    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b033e2-ebcc-4112-99d6-d9398e5ac835-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.055667 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.055652    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tc4\" (UniqueName: \"kubernetes.io/projected/e5b033e2-ebcc-4112-99d6-d9398e5ac835-kube-api-access-w6tc4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-bg7cl\" (UID: \"e5b033e2-ebcc-4112-99d6-d9398e5ac835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.088408 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.088381    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:54.362699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.362671    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"]
Nov 03 19:40:54.365374 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:40:54.365340    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b033e2_ebcc_4112_99d6_d9398e5ac835.slice/crio-59fbe0cda0fc9486762d46100f66e054b6d181902a764ec3a7b2d8b660d7a856 WatchSource:0}: Error finding container 59fbe0cda0fc9486762d46100f66e054b6d181902a764ec3a7b2d8b660d7a856: Status 404 returned error can't find the container with id 59fbe0cda0fc9486762d46100f66e054b6d181902a764ec3a7b2d8b660d7a856
Nov 03 19:40:54.370589 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.370546    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:54.370711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.370635    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:54.370711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:54.370678    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:55.167042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:55.167006    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl" event={"ID":"e5b033e2-ebcc-4112-99d6-d9398e5ac835","Type":"ContainerStarted","Data":"611ca75a205bc07818b3d0bc5e29bc7d22588327adcafb84f3292c9dc2eb1d83"}
Nov 03 19:40:55.167235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:55.167061    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl" event={"ID":"e5b033e2-ebcc-4112-99d6-d9398e5ac835","Type":"ContainerStarted","Data":"59fbe0cda0fc9486762d46100f66e054b6d181902a764ec3a7b2d8b660d7a856"}
Nov 03 19:40:55.185233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:55.185018    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl" podStartSLOduration=2.185001567 podStartE2EDuration="2.185001567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:55.184797611 +0000 UTC m=+568.454920904" watchObservedRunningTime="2026-11-03 19:40:55.185001567 +0000 UTC m=+568.455124860"
Nov 03 19:40:56.088628 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:56.088585    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:40:56.090221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:56.090194    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 19:40:56.090343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:56.090241    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl" podUID="e5b033e2-ebcc-4112-99d6-d9398e5ac835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 19:40:57.089320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:57.089293    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 19:40:57.089696 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:57.089340    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl" podUID="e5b033e2-ebcc-4112-99d6-d9398e5ac835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 19:40:58.088996 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:58.088960    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 19:40:58.089207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:58.089011    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl" podUID="e5b033e2-ebcc-4112-99d6-d9398e5ac835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 19:40:59.089532 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:59.089490    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 19:40:59.089931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:40:59.089583    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl" podUID="e5b033e2-ebcc-4112-99d6-d9398e5ac835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 19:41:00.089396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:00.089356    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 19:41:00.089603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:00.089429    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl" podUID="e5b033e2-ebcc-4112-99d6-d9398e5ac835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 19:41:01.089357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:01.089317    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 19:41:01.089542 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:01.089398    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl" podUID="e5b033e2-ebcc-4112-99d6-d9398e5ac835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 19:41:02.472101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:02.472067    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:41:02.472605 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:02.472561    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:41:02.473363 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:02.473342    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl"
Nov 03 19:41:04.153772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:04.153744    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-9bf4fd58b-7rs7s"
Nov 03 19:41:04.194107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:04.194053    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-tk99k"]
Nov 03 19:41:04.194635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:04.194598    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k" podUID="ae7a981e-d31b-4410-9502-5d7a90f381e9" containerName="manager" containerID="cri-o://de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616" gracePeriod=30
Nov 03 19:41:04.445791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:04.445768    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:41:04.557167 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:04.557136    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae7a981e-d31b-4410-9502-5d7a90f381e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7a981e-d31b-4410-9502-5d7a90f381e9-metrics-certs\") pod \"ae7a981e-d31b-4410-9502-5d7a90f381e9\" (UID: \"ae7a981e-d31b-4410-9502-5d7a90f381e9\") "
Nov 03 19:41:04.557357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:04.557210    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7a981e-d31b-4410-9502-5d7a90f381e9-kube-api-access-96cml\" (UniqueName: \"kubernetes.io/projected/ae7a981e-d31b-4410-9502-5d7a90f381e9-kube-api-access-96cml\") pod \"ae7a981e-d31b-4410-9502-5d7a90f381e9\" (UID: \"ae7a981e-d31b-4410-9502-5d7a90f381e9\") "
Nov 03 19:41:04.559444 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:04.559412    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7a981e-d31b-4410-9502-5d7a90f381e9-kube-api-access-96cml" pod "ae7a981e-d31b-4410-9502-5d7a90f381e9" (UID: "ae7a981e-d31b-4410-9502-5d7a90f381e9"). InnerVolumeSpecName "kube-api-access-96cml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:04.559564 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:04.559471    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7a981e-d31b-4410-9502-5d7a90f381e9-metrics-certs" pod "ae7a981e-d31b-4410-9502-5d7a90f381e9" (UID: "ae7a981e-d31b-4410-9502-5d7a90f381e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:41:04.657842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:04.657811    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7a981e-d31b-4410-9502-5d7a90f381e9-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:04.657842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:04.657837    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96cml\" (UniqueName: \"kubernetes.io/projected/ae7a981e-d31b-4410-9502-5d7a90f381e9-kube-api-access-96cml\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:05.218416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:05.218384    4531 generic.go:356] "Generic (PLEG): container finished" podID="ae7a981e-d31b-4410-9502-5d7a90f381e9" containerID="de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616" exitCode=0
Nov 03 19:41:05.218842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:05.218428    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k"
Nov 03 19:41:05.218842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:05.218455    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k" event={"ID":"ae7a981e-d31b-4410-9502-5d7a90f381e9","Type":"ContainerDied","Data":"de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616"}
Nov 03 19:41:05.218842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:05.218498    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-tk99k" event={"ID":"ae7a981e-d31b-4410-9502-5d7a90f381e9","Type":"ContainerDied","Data":"d0e9b4a66498ca1b1e682357e0ced7d9ac90bf7121db40ad0dbb6577f25bec18"}
Nov 03 19:41:05.218842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:05.218521    4531 scope.go:122] "RemoveContainer" containerID="de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616"
Nov 03 19:41:05.226997 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:05.226972    4531 scope.go:122] "RemoveContainer" containerID="de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616"
Nov 03 19:41:05.227271 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:41:05.227244    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616\": container with ID starting with de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616 not found: ID does not exist" containerID="de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616"
Nov 03 19:41:05.227362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:05.227293    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616"} err="failed to get container status \"de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616\": rpc error: code = NotFound desc = could not find container \"de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616\": container with ID starting with de628bb50f9a8b0b2c218395ff3bd0f488628e4a674b9daeb4e8725cdbe38616 not found: ID does not exist"
Nov 03 19:41:05.232647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:05.232625    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-tk99k"]
Nov 03 19:41:05.236424 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:05.236403    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-tk99k"]
Nov 03 19:41:05.804807 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:05.804773    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:06.856275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:06.856243    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7a981e-d31b-4410-9502-5d7a90f381e9" path="/var/lib/kubelet/pods/ae7a981e-d31b-4410-9502-5d7a90f381e9/volumes"
Nov 03 19:41:07.496459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:07.496427    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:07.792604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:07.792522    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:07.940108 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:07.940064    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:08.799648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.799615    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"]
Nov 03 19:41:08.800292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.800264    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ae7a981e-d31b-4410-9502-5d7a90f381e9" containerName="manager"
Nov 03 19:41:08.800292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.800288    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7a981e-d31b-4410-9502-5d7a90f381e9" containerName="manager"
Nov 03 19:41:08.800474 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.800397    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7a981e-d31b-4410-9502-5d7a90f381e9" containerName="manager"
Nov 03 19:41:08.807158 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.807137    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.812365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.812339    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"]
Nov 03 19:41:08.896471 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.896439    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.896647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.896520    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.896647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.896631    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.896774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.896670    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.896774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.896701    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4287\" (UniqueName: \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-kube-api-access-d4287\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.896774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.896749    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.896928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.896778    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.896928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.896806    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.896928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.896834    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.997872 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.997838    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4287\" (UniqueName: \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-kube-api-access-d4287\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.997872 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.997877    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.998435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.997902    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.998435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.997929    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.998435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.997954    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.998435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.998020    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.998435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.998132    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.998435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.998214    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.998435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.998260    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.998435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.998414    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.998843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.998438    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.998843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.998803    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.998843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.998829    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:08.999126 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:08.999098    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:09.000536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:09.000514    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:09.000799 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:09.000780    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:09.004508 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:09.004488    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:09.004895 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:09.004875    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4287\" (UniqueName: \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-kube-api-access-d4287\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-g74zd\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:09.129361 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:09.129336    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:09.214651 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:09.214622    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:09.358815 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:09.358782    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:09.407690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:09.407666    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"]
Nov 03 19:41:09.410051 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:41:09.410019    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3fc4ac_760e_444b_a9eb_967f4d443b8e.slice/crio-f010dc96bd081d990bd8fbb2725ae8e515c1e4181ea2c1a77714ff69eb6290aa WatchSource:0}: Error finding container f010dc96bd081d990bd8fbb2725ae8e515c1e4181ea2c1a77714ff69eb6290aa: Status 404 returned error can't find the container with id f010dc96bd081d990bd8fbb2725ae8e515c1e4181ea2c1a77714ff69eb6290aa
Nov 03 19:41:09.412160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:09.412128    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:09.412266 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:09.412221    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:09.412322 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:09.412279    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:10.263348 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:10.263296    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" event={"ID":"6e3fc4ac-760e-444b-a9eb-967f4d443b8e","Type":"ContainerStarted","Data":"ac9467ce31127da3b59ab80b4abd00f54c3c38e37a395bc35d8f752eab378170"}
Nov 03 19:41:10.263861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:10.263376    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" event={"ID":"6e3fc4ac-760e-444b-a9eb-967f4d443b8e","Type":"ContainerStarted","Data":"f010dc96bd081d990bd8fbb2725ae8e515c1e4181ea2c1a77714ff69eb6290aa"}
Nov 03 19:41:10.280166 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:10.280089    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" podStartSLOduration=2.280073089 podStartE2EDuration="2.280073089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:10.279277596 +0000 UTC m=+583.549400884" watchObservedRunningTime="2026-11-03 19:41:10.280073089 +0000 UTC m=+583.550196383"
Nov 03 19:41:11.129685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:11.129652    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:11.131271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:11.131247    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-g74zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 19:41:11.131411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:11.131304    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 19:41:12.129788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:12.129756    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-g74zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 19:41:12.130195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:12.129808    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 19:41:13.129760 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:13.129720    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-g74zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 19:41:13.129945 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:13.129815    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 19:41:14.129730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:14.129694    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-g74zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 19:41:14.129888 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:14.129751    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 19:41:15.078773 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:15.078740    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:15.130693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:15.130664    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-g74zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 19:41:15.130823 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:15.130716    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 19:41:16.130344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:16.130298    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-g74zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 19:41:16.130968 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:16.130362    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 19:41:17.481148 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:17.481115    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:17.482108 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:17.482083    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:17.483257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:17.483232    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 19:41:20.571616 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:41:20.571579    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:41:26.906009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:26.905560    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:41:26.908818 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:26.908788    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:41:31.718490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:31.718429    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:31.843126 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:31.843087    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:32.047198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.047099    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"]
Nov 03 19:41:32.059163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.059126    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.064360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.064331    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"]
Nov 03 19:41:32.069341 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.069317    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8d4127-openshift-default-dockercfg-6m7xm\""
Nov 03 19:41:32.144887 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.144850    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.145127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.144905    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.145127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.144983    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.145127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.145030    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.145127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.145051    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.145127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.145074    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.145459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.145144    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mz2x\" (UniqueName: \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-kube-api-access-2mz2x\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.145459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.145216    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.145459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.145239    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.169204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.169149    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:32.246794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.246749    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.246984 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.246811    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.246984 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.246848    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.246984 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.246924    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mz2x\" (UniqueName: \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-kube-api-access-2mz2x\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.247192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.247072    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.247192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.247107    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.247192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.247166    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.247374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.247274    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.247374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.247303    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.247819 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.247776    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.248606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.248585    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.248796 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.248766    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.249244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.249220    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.249642 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.249621    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.257153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.257117    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mz2x\" (UniqueName: \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-kube-api-access-2mz2x\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.258052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.258020    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.261598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.261575    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.261796 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.261773    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.324209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.324118    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"]
Nov 03 19:41:32.328832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.328808    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.339394 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.339342    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"]
Nov 03 19:41:32.348758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.348732    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.348891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.348766    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7dhg\" (UniqueName: \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-kube-api-access-k7dhg\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.348891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.348789    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.348891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.348872    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.349028 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.348918    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.349028 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.348949    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.349096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.349039    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.349096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.349065    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.349219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.349094    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.354269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.354246    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:32.371444 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.371419    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-82dbbf-openshift-default-dockercfg-wz5bm\""
Nov 03 19:41:32.380237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.380201    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:32.450051 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.450021    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.450235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.450082    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.450235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.450118    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.450235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.450144    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.450235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.450192    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.450458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.450261    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.450458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.450294    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dhg\" (UniqueName: \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-kube-api-access-k7dhg\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.450458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.450328    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.450458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.450360    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.450650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.450486    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.450843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.450820    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.451093 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.451059    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.451611 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.451586    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.451855 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.451836    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.454323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.454297    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.454681 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.454657    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.479038 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.478985    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7dhg\" (UniqueName: \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-kube-api-access-k7dhg\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.479321 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.479298    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.495290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.495249    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"]
Nov 03 19:41:32.501417 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.501384    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.517629 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.517599    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f4e123-openshift-default-dockercfg-qlw2x\""
Nov 03 19:41:32.521749 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.521710    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"]
Nov 03 19:41:32.551450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.551418    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.551593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.551454    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.551593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.551566    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2bcz\" (UniqueName: \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-kube-api-access-m2bcz\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.551692 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.551593    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdaf9c8-7009-4378-8f63-3071e17f5525-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.551692 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.551616    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.551773 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.551710    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.551773 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.551762    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.551849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.551800    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.551884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.551861    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.631163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.631126    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:32.647916 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.647890    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:32.653362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.653337    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.653478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.653400    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.653478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.653471    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.653604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.653500    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.653604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.653565    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2bcz\" (UniqueName: \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-kube-api-access-m2bcz\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.653604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.653588    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdaf9c8-7009-4378-8f63-3071e17f5525-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.653870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.653850    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.654004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.653900    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.654004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.653942    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.654596 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.654361    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.654596 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.654452    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.655338 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.655315    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.655445 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.655341    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.655785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.655764    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdaf9c8-7009-4378-8f63-3071e17f5525-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.656573 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.656557    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.657202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.657186    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.670040 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.666462    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2bcz\" (UniqueName: \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-kube-api-access-m2bcz\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.671109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.671066    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.672837 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.672758    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"]
Nov 03 19:41:32.687861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.687822    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:32.687959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.687915    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:32.688018 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.687989    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:32.782661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.782625    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:32.830316 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.829316    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:32.934861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.933772    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:32.963600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.963563    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"]
Nov 03 19:41:32.969394 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:41:32.969358    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2ffbba_e334_4eb9_b86b_184fc8c04bed.slice/crio-5c6c83fee64dd39c3d52b8a470cc85687fdc4602ca9b7a5e171c1bafc234b292 WatchSource:0}: Error finding container 5c6c83fee64dd39c3d52b8a470cc85687fdc4602ca9b7a5e171c1bafc234b292: Status 404 returned error can't find the container with id 5c6c83fee64dd39c3d52b8a470cc85687fdc4602ca9b7a5e171c1bafc234b292
Nov 03 19:41:32.976656 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.976614    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:32.976739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.976702    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:32.976794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:32.976753    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:33.081365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.081331    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:33.200948 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.200923    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"]
Nov 03 19:41:33.203708 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:41:33.203674    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdaf9c8_7009_4378_8f63_3071e17f5525.slice/crio-7af75fdcd4e86964ad58d0af66bcf439343db668e1aa25635c9280562211f204 WatchSource:0}: Error finding container 7af75fdcd4e86964ad58d0af66bcf439343db668e1aa25635c9280562211f204: Status 404 returned error can't find the container with id 7af75fdcd4e86964ad58d0af66bcf439343db668e1aa25635c9280562211f204
Nov 03 19:41:33.206036 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.206005    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:33.206124 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.206076    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:33.206124 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.206107    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:33.366248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.366210    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" event={"ID":"e68ef5cf-00e6-423c-92f6-ff8200f459b1","Type":"ContainerStarted","Data":"9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004"}
Nov 03 19:41:33.366421 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.366277    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" event={"ID":"e68ef5cf-00e6-423c-92f6-ff8200f459b1","Type":"ContainerStarted","Data":"f179c44b64d0d04e7e1e556c09fd58f67db5e4f1e5835378fec9215e613fe54a"}
Nov 03 19:41:33.368700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.368666    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" event={"ID":"fcdaf9c8-7009-4378-8f63-3071e17f5525","Type":"ContainerStarted","Data":"89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547"}
Nov 03 19:41:33.368828 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.368732    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" event={"ID":"fcdaf9c8-7009-4378-8f63-3071e17f5525","Type":"ContainerStarted","Data":"7af75fdcd4e86964ad58d0af66bcf439343db668e1aa25635c9280562211f204"}
Nov 03 19:41:33.370967 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.370942    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" event={"ID":"aa2ffbba-e334-4eb9-b86b-184fc8c04bed","Type":"ContainerStarted","Data":"8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4"}
Nov 03 19:41:33.371072 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.370987    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" event={"ID":"aa2ffbba-e334-4eb9-b86b-184fc8c04bed","Type":"ContainerStarted","Data":"5c6c83fee64dd39c3d52b8a470cc85687fdc4602ca9b7a5e171c1bafc234b292"}
Nov 03 19:41:33.381346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.381322    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:33.386890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.386856    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:33.389802 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.389752    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" podStartSLOduration=1.389737585 podStartE2EDuration="1.389737585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:33.38832121 +0000 UTC m=+606.658444505" watchObservedRunningTime="2026-11-03 19:41:33.389737585 +0000 UTC m=+606.659860880"
Nov 03 19:41:33.410137 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.410057    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" podStartSLOduration=1.41003885 podStartE2EDuration="1.41003885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:33.408891156 +0000 UTC m=+606.679014468" watchObservedRunningTime="2026-11-03 19:41:33.41003885 +0000 UTC m=+606.680162151"
Nov 03 19:41:33.425274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.425216    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" podStartSLOduration=1.425199552 podStartE2EDuration="1.425199552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:33.424896304 +0000 UTC m=+606.695019599" watchObservedRunningTime="2026-11-03 19:41:33.425199552 +0000 UTC m=+606.695322848"
Nov 03 19:41:33.648745 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.648709    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:33.830293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:33.830256    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:34.376165 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:34.376124    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:34.376910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:34.376892    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:41:34.653525 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:34.653499    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:34.834893 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:34.834863    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:35.380878 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:35.380843    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:35.381056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:35.380895    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:35.381538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:35.381518    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:35.381643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:35.381568    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:41:35.494534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:35.494505    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:35.646397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:35.646305    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:35.815537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:35.815502    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:36.085139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:36.085049    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:36.274418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:36.274389    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:36.434564 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:36.434536    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:36.574509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:36.574477    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:36.963535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:36.963498    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:37.111717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.111681    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:37.672959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.672919    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:37.678391 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.678361    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"]
Nov 03 19:41:37.682383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.682345    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 19:41:37.688254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.688158    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a35658\""
Nov 03 19:41:37.690056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.690034    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"]
Nov 03 19:41:37.711050 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.711022    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a35658-dockercfg-bzh4s\""
Nov 03 19:41:37.750664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.750621    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmw7v\" (UniqueName: \"kubernetes.io/projected/b511ca1a-779c-45e1-b233-55b34ba02945-kube-api-access-bmw7v\") pod \"payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4\" (UID: \"b511ca1a-779c-45e1-b233-55b34ba02945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 19:41:37.750841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.750805    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b511ca1a-779c-45e1-b233-55b34ba02945-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4\" (UID: \"b511ca1a-779c-45e1-b233-55b34ba02945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 19:41:37.815054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.814995    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:37.851559 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.851525    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b511ca1a-779c-45e1-b233-55b34ba02945-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4\" (UID: \"b511ca1a-779c-45e1-b233-55b34ba02945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 19:41:37.851762 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.851615    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmw7v\" (UniqueName: \"kubernetes.io/projected/b511ca1a-779c-45e1-b233-55b34ba02945-kube-api-access-bmw7v\") pod \"payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4\" (UID: \"b511ca1a-779c-45e1-b233-55b34ba02945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 19:41:37.852410 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.852382    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b511ca1a-779c-45e1-b233-55b34ba02945-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4\" (UID: \"b511ca1a-779c-45e1-b233-55b34ba02945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 19:41:37.859543 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.859511    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmw7v\" (UniqueName: \"kubernetes.io/projected/b511ca1a-779c-45e1-b233-55b34ba02945-kube-api-access-bmw7v\") pod \"payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4\" (UID: \"b511ca1a-779c-45e1-b233-55b34ba02945\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 19:41:37.998161 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:37.998061    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 19:41:38.210111 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.210076    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:38.270127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.270033    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"]
Nov 03 19:41:38.273514 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:41:38.273483    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb511ca1a_779c_45e1_b233_55b34ba02945.slice/crio-5faa00ec42cb25a9ca95524179e56000b748121fbccbffc88b2048f972760996 WatchSource:0}: Error finding container 5faa00ec42cb25a9ca95524179e56000b748121fbccbffc88b2048f972760996: Status 404 returned error can't find the container with id 5faa00ec42cb25a9ca95524179e56000b748121fbccbffc88b2048f972760996
Nov 03 19:41:38.275800 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.275781    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:41:38.347066 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.346963    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:38.396938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.396902    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4" event={"ID":"b511ca1a-779c-45e1-b233-55b34ba02945","Type":"ContainerStarted","Data":"5faa00ec42cb25a9ca95524179e56000b748121fbccbffc88b2048f972760996"}
Nov 03 19:41:38.718267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.718235    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"]
Nov 03 19:41:38.723230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.723189    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 19:41:38.724375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.724347    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:38.728520 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.728491    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"]
Nov 03 19:41:38.729153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.729133    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f4e123\""
Nov 03 19:41:38.732667 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.732647    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f4e123-dockercfg-ml6kn\""
Nov 03 19:41:38.740271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.740246    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"]
Nov 03 19:41:38.744602 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.744578    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 19:41:38.755447 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.755412    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"]
Nov 03 19:41:38.863763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.863731    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4smrq\" (UniqueName: \"kubernetes.io/projected/752cf009-4d23-4820-ae58-b3ad21317e57-kube-api-access-4smrq\") pod \"payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94\" (UID: \"752cf009-4d23-4820-ae58-b3ad21317e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 19:41:38.864072 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.864051    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752cf009-4d23-4820-ae58-b3ad21317e57-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94\" (UID: \"752cf009-4d23-4820-ae58-b3ad21317e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 19:41:38.864285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.864265    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db869375-d3d6-4043-b87f-5df736ed2939-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m\" (UID: \"db869375-d3d6-4043-b87f-5df736ed2939\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 19:41:38.864435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.864420    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qspsd\" (UniqueName: \"kubernetes.io/projected/db869375-d3d6-4043-b87f-5df736ed2939-kube-api-access-qspsd\") pod \"payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m\" (UID: \"db869375-d3d6-4043-b87f-5df736ed2939\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 19:41:38.905203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.896398    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:38.965478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.965434    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db869375-d3d6-4043-b87f-5df736ed2939-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m\" (UID: \"db869375-d3d6-4043-b87f-5df736ed2939\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 19:41:38.965716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.965504    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qspsd\" (UniqueName: \"kubernetes.io/projected/db869375-d3d6-4043-b87f-5df736ed2939-kube-api-access-qspsd\") pod \"payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m\" (UID: \"db869375-d3d6-4043-b87f-5df736ed2939\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 19:41:38.965716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.965597    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smrq\" (UniqueName: \"kubernetes.io/projected/752cf009-4d23-4820-ae58-b3ad21317e57-kube-api-access-4smrq\") pod \"payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94\" (UID: \"752cf009-4d23-4820-ae58-b3ad21317e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 19:41:38.965716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.965695    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752cf009-4d23-4820-ae58-b3ad21317e57-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94\" (UID: \"752cf009-4d23-4820-ae58-b3ad21317e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 19:41:38.966369 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.966349    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db869375-d3d6-4043-b87f-5df736ed2939-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m\" (UID: \"db869375-d3d6-4043-b87f-5df736ed2939\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 19:41:38.966626 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.966589    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752cf009-4d23-4820-ae58-b3ad21317e57-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94\" (UID: \"752cf009-4d23-4820-ae58-b3ad21317e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 19:41:38.974881 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.974818    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qspsd\" (UniqueName: \"kubernetes.io/projected/db869375-d3d6-4043-b87f-5df736ed2939-kube-api-access-qspsd\") pod \"payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m\" (UID: \"db869375-d3d6-4043-b87f-5df736ed2939\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 19:41:38.977198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:38.977148    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4smrq\" (UniqueName: \"kubernetes.io/projected/752cf009-4d23-4820-ae58-b3ad21317e57-kube-api-access-4smrq\") pod \"payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94\" (UID: \"752cf009-4d23-4820-ae58-b3ad21317e57\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 19:41:39.039455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.039422    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 19:41:39.060523 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.060483    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 19:41:39.177610 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.177563    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:39.244771 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.244685    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"]
Nov 03 19:41:39.250056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.250031    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 19:41:39.258969 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.257391    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"]
Nov 03 19:41:39.307485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.307455    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-8d4127-dockercfg-jdf88\""
Nov 03 19:41:39.318093 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.318057    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-8d4127\""
Nov 03 19:41:39.325458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.325430    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:39.337391 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.337360    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"]
Nov 03 19:41:39.384015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.382659    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cd9h\" (UniqueName: \"kubernetes.io/projected/40903bcb-9b64-4233-8230-5b74407797fb-kube-api-access-2cd9h\") pod \"payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58\" (UID: \"40903bcb-9b64-4233-8230-5b74407797fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 19:41:39.384015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.382803    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40903bcb-9b64-4233-8230-5b74407797fb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58\" (UID: \"40903bcb-9b64-4233-8230-5b74407797fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 19:41:39.401220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.399204    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"]
Nov 03 19:41:39.425476 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:41:39.425113    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752cf009_4d23_4820_ae58_b3ad21317e57.slice/crio-2f3f17fc22e9ef5d803e68b291007c9ed43d99e6ff18e084ef8091236ffc3a1a WatchSource:0}: Error finding container 2f3f17fc22e9ef5d803e68b291007c9ed43d99e6ff18e084ef8091236ffc3a1a: Status 404 returned error can't find the container with id 2f3f17fc22e9ef5d803e68b291007c9ed43d99e6ff18e084ef8091236ffc3a1a
Nov 03 19:41:39.453912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.453689    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:39.461541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.461464    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m" event={"ID":"db869375-d3d6-4043-b87f-5df736ed2939","Type":"ContainerStarted","Data":"aa337e0db6f82a117e3bf27e49b16ce5fc506764559cc0027606f9dd10bcb134"}
Nov 03 19:41:39.483806 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.483755    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cd9h\" (UniqueName: \"kubernetes.io/projected/40903bcb-9b64-4233-8230-5b74407797fb-kube-api-access-2cd9h\") pod \"payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58\" (UID: \"40903bcb-9b64-4233-8230-5b74407797fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 19:41:39.484146 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.483868    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40903bcb-9b64-4233-8230-5b74407797fb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58\" (UID: \"40903bcb-9b64-4233-8230-5b74407797fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 19:41:39.484736 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.484711    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40903bcb-9b64-4233-8230-5b74407797fb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58\" (UID: \"40903bcb-9b64-4233-8230-5b74407797fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 19:41:39.491505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.491466    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cd9h\" (UniqueName: \"kubernetes.io/projected/40903bcb-9b64-4233-8230-5b74407797fb-kube-api-access-2cd9h\") pod \"payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58\" (UID: \"40903bcb-9b64-4233-8230-5b74407797fb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 19:41:39.576382 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.576303    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 19:41:39.680949 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.680866    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:39.696991 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.696959    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"]
Nov 03 19:41:39.702579 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.702549    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 19:41:39.709910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.709882    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"]
Nov 03 19:41:39.716304 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.716070    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-82dbbf\""
Nov 03 19:41:39.716304 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.716086    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-82dbbf-dockercfg-67nc6\""
Nov 03 19:41:39.787793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.787751    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0507af87-0ab8-46b3-9c18-45a7af463894-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5\" (UID: \"0507af87-0ab8-46b3-9c18-45a7af463894\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 19:41:39.787946 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.787808    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b4nx\" (UniqueName: \"kubernetes.io/projected/0507af87-0ab8-46b3-9c18-45a7af463894-kube-api-access-6b4nx\") pod \"payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5\" (UID: \"0507af87-0ab8-46b3-9c18-45a7af463894\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 19:41:39.828663 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.828567    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:39.881691 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.881657    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"]
Nov 03 19:41:39.891110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.891056    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0507af87-0ab8-46b3-9c18-45a7af463894-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5\" (UID: \"0507af87-0ab8-46b3-9c18-45a7af463894\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 19:41:39.891336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.891124    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b4nx\" (UniqueName: \"kubernetes.io/projected/0507af87-0ab8-46b3-9c18-45a7af463894-kube-api-access-6b4nx\") pod \"payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5\" (UID: \"0507af87-0ab8-46b3-9c18-45a7af463894\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 19:41:39.892399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.892342    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0507af87-0ab8-46b3-9c18-45a7af463894-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5\" (UID: \"0507af87-0ab8-46b3-9c18-45a7af463894\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 19:41:39.924111 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:39.921379    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b4nx\" (UniqueName: \"kubernetes.io/projected/0507af87-0ab8-46b3-9c18-45a7af463894-kube-api-access-6b4nx\") pod \"payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5\" (UID: \"0507af87-0ab8-46b3-9c18-45a7af463894\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 19:41:40.022935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.022900    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 19:41:40.143864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.143778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:40.269976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.269900    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:40.345751 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.345671    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"]
Nov 03 19:41:40.402378 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:41:40.401545    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0507af87_0ab8_46b3_9c18_45a7af463894.slice/crio-cdd14945467d2b34e53ae2ad8a991255140782b6a63465c3575cefc26344b9a0 WatchSource:0}: Error finding container cdd14945467d2b34e53ae2ad8a991255140782b6a63465c3575cefc26344b9a0: Status 404 returned error can't find the container with id cdd14945467d2b34e53ae2ad8a991255140782b6a63465c3575cefc26344b9a0
Nov 03 19:41:40.402902 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.402793    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:40.473395 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.473310    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94" event={"ID":"752cf009-4d23-4820-ae58-b3ad21317e57","Type":"ContainerStarted","Data":"2f3f17fc22e9ef5d803e68b291007c9ed43d99e6ff18e084ef8091236ffc3a1a"}
Nov 03 19:41:40.475503 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.475409    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5" event={"ID":"0507af87-0ab8-46b3-9c18-45a7af463894","Type":"ContainerStarted","Data":"cdd14945467d2b34e53ae2ad8a991255140782b6a63465c3575cefc26344b9a0"}
Nov 03 19:41:40.476968 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.476900    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58" event={"ID":"40903bcb-9b64-4233-8230-5b74407797fb","Type":"ContainerStarted","Data":"4f58756c1fcdb192250b1848f75a5ca613666c4754146eb5ea1486c2ee8473c2"}
Nov 03 19:41:40.491665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.491632    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"]
Nov 03 19:41:40.495431 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.495404    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 19:41:40.504531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.503852    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-0486e8-dockercfg-rspkc\""
Nov 03 19:41:40.504531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.503943    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-0486e8\""
Nov 03 19:41:40.507946 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.507694    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"]
Nov 03 19:41:40.533298 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.533150    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"]
Nov 03 19:41:40.537302 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.537273    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 19:41:40.545396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.545344    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"]
Nov 03 19:41:40.548264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.548203    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:40.628798 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.628718    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g6jw\" (UniqueName: \"kubernetes.io/projected/72c91d62-295e-497e-a507-6244fa438caa-kube-api-access-6g6jw\") pod \"payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj\" (UID: \"72c91d62-295e-497e-a507-6244fa438caa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 19:41:40.628989 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.628951    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c91d62-295e-497e-a507-6244fa438caa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj\" (UID: \"72c91d62-295e-497e-a507-6244fa438caa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 19:41:40.629041 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.629026    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvghb\" (UniqueName: \"kubernetes.io/projected/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-kube-api-access-xvghb\") pod \"payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg\" (UID: \"f596e70f-a49c-47e8-8b8d-e5d27612f5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 19:41:40.629093 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.629077    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg\" (UID: \"f596e70f-a49c-47e8-8b8d-e5d27612f5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 19:41:40.743709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.741781    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g6jw\" (UniqueName: \"kubernetes.io/projected/72c91d62-295e-497e-a507-6244fa438caa-kube-api-access-6g6jw\") pod \"payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj\" (UID: \"72c91d62-295e-497e-a507-6244fa438caa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 19:41:40.743709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.741914    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c91d62-295e-497e-a507-6244fa438caa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj\" (UID: \"72c91d62-295e-497e-a507-6244fa438caa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 19:41:40.743709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.741968    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvghb\" (UniqueName: \"kubernetes.io/projected/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-kube-api-access-xvghb\") pod \"payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg\" (UID: \"f596e70f-a49c-47e8-8b8d-e5d27612f5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 19:41:40.743709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.742015    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg\" (UID: \"f596e70f-a49c-47e8-8b8d-e5d27612f5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 19:41:40.743709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.742976    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg\" (UID: \"f596e70f-a49c-47e8-8b8d-e5d27612f5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 19:41:40.744924 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.744896    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c91d62-295e-497e-a507-6244fa438caa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj\" (UID: \"72c91d62-295e-497e-a507-6244fa438caa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 19:41:40.788840 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.788467    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g6jw\" (UniqueName: \"kubernetes.io/projected/72c91d62-295e-497e-a507-6244fa438caa-kube-api-access-6g6jw\") pod \"payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj\" (UID: \"72c91d62-295e-497e-a507-6244fa438caa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 19:41:40.790004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.789972    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvghb\" (UniqueName: \"kubernetes.io/projected/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-kube-api-access-xvghb\") pod \"payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg\" (UID: \"f596e70f-a49c-47e8-8b8d-e5d27612f5c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 19:41:40.832954 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.832750    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 19:41:40.870665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:40.870626    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 19:41:41.037449 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:41.037377    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:41.188944 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:41.188861    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"]
Nov 03 19:41:41.286687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:41.286657    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"]
Nov 03 19:41:41.296872 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:41:41.296817    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf596e70f_a49c_47e8_8b8d_e5d27612f5c7.slice/crio-93d65541e3444a25ccceed090d6420cee0bddf3d39d2f03b9b4c457cf6f4b6f3 WatchSource:0}: Error finding container 93d65541e3444a25ccceed090d6420cee0bddf3d39d2f03b9b4c457cf6f4b6f3: Status 404 returned error can't find the container with id 93d65541e3444a25ccceed090d6420cee0bddf3d39d2f03b9b4c457cf6f4b6f3
Nov 03 19:41:41.493803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:41.493709    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg" event={"ID":"f596e70f-a49c-47e8-8b8d-e5d27612f5c7","Type":"ContainerStarted","Data":"93d65541e3444a25ccceed090d6420cee0bddf3d39d2f03b9b4c457cf6f4b6f3"}
Nov 03 19:41:41.496989 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:41.496949    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj" event={"ID":"72c91d62-295e-497e-a507-6244fa438caa","Type":"ContainerStarted","Data":"e41a554f6c9a9140a93578fb5ce89cc12f1ffa85573ca7502558ec9b73ab2c4f"}
Nov 03 19:41:41.500781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:41.500675    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:41.696796 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:41.696758    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:41.923693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:41.923569    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:42.054979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.054856    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:42.200325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.200253    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:42.757079 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.756308    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"]
Nov 03 19:41:42.763033 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.761712    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.772193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.768853    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:41:42.773581 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.773554    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"]
Nov 03 19:41:42.783826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.774656    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:41:42.784026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.774700    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rj6j4\""
Nov 03 19:41:42.784026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.774706    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-82dbbf-serving-cert\""
Nov 03 19:41:42.784026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.783367    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 19:41:42.814754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.814712    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-maas-api-tls\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.815091 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.815064    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhd2g\" (UniqueName: \"kubernetes.io/projected/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-kube-api-access-qhd2g\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.815315 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.815232    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.815409 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.815388    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-metrics-certs\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.857054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.856983    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:42.916627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.916573    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.916835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.916656    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-metrics-certs\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.916835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.916716    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-maas-api-tls\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.916835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.916744    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhd2g\" (UniqueName: \"kubernetes.io/projected/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-kube-api-access-qhd2g\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.917757 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.917732    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.939250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.939105    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-maas-api-tls\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.939842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.939790    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-metrics-certs\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.949947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.949898    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhd2g\" (UniqueName: \"kubernetes.io/projected/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-kube-api-access-qhd2g\") pod \"maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:42.993040 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:42.992957    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:43.095790 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:43.095709    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:43.134770 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:43.134670    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:43.484696 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:43.484610    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"]
Nov 03 19:41:43.545511 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:41:43.545472    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea0c8fc_05ab_4173_8084_ad7e2bc678ad.slice/crio-547cc54f5d8d7adc2de88fa87e06278c9b79349998d739ed352cd090f4b0c84a WatchSource:0}: Error finding container 547cc54f5d8d7adc2de88fa87e06278c9b79349998d739ed352cd090f4b0c84a: Status 404 returned error can't find the container with id 547cc54f5d8d7adc2de88fa87e06278c9b79349998d739ed352cd090f4b0c84a
Nov 03 19:41:43.761353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:43.761228    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:44.563777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.563724    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4" event={"ID":"b511ca1a-779c-45e1-b233-55b34ba02945","Type":"ContainerStarted","Data":"a5e566870b104275c241ab2d21a85e6c9ecb3d13c06abd841be76dc6222af1a9"}
Nov 03 19:41:44.564952 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.564562    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 19:41:44.567266 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.567241    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg" event={"ID":"f596e70f-a49c-47e8-8b8d-e5d27612f5c7","Type":"ContainerStarted","Data":"52466bd853f4d7f9c59844944a312f5f0d6ecc0d2d13715d02ff1ce0cfb1fc16"}
Nov 03 19:41:44.567386 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.567374    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 19:41:44.573022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.572401    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj" event={"ID":"72c91d62-295e-497e-a507-6244fa438caa","Type":"ContainerStarted","Data":"02fd31d8e5afee285f86496f6e597632374cb4bdac1e4a6d27b43e1aaa511623"}
Nov 03 19:41:44.573022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.572951    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 19:41:44.580907 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.580883    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5" event={"ID":"0507af87-0ab8-46b3-9c18-45a7af463894","Type":"ContainerStarted","Data":"c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03"}
Nov 03 19:41:44.581523 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.581496    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 19:41:44.584793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.582452    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4" podStartSLOduration=1.956143572 podStartE2EDuration="7.582436127s" totalImagesPullingTime="5.626292555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:44.58086667 +0000 UTC m=+617.850989966" watchObservedRunningTime="2026-11-03 19:41:44.582436127 +0000 UTC m=+617.852559423"
Nov 03 19:41:44.584793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.583952    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz" event={"ID":"dea0c8fc-05ab-4173-8084-ad7e2bc678ad","Type":"ContainerStarted","Data":"547cc54f5d8d7adc2de88fa87e06278c9b79349998d739ed352cd090f4b0c84a"}
Nov 03 19:41:44.584793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.584223    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:44.598370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.597489    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg" podStartSLOduration=1.555275983 podStartE2EDuration="4.597471392s" totalImagesPullingTime="3.042195409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:44.593850304 +0000 UTC m=+617.863973601" watchObservedRunningTime="2026-11-03 19:41:44.597471392 +0000 UTC m=+617.867594690"
Nov 03 19:41:44.615466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.613222    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj" podStartSLOduration=1.438603134 podStartE2EDuration="4.613205221s" totalImagesPullingTime="3.174602087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:44.609921628 +0000 UTC m=+617.880044924" watchObservedRunningTime="2026-11-03 19:41:44.613205221 +0000 UTC m=+617.883328517"
Nov 03 19:41:44.628683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.628622    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5" podStartSLOduration=1.786354095 podStartE2EDuration="5.628605256s" totalImagesPullingTime="3.842251161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:44.625022224 +0000 UTC m=+617.895145519" watchObservedRunningTime="2026-11-03 19:41:44.628605256 +0000 UTC m=+617.898728553"
Nov 03 19:41:44.711805 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:44.711697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:45.592874 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:45.592738    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m" event={"ID":"db869375-d3d6-4043-b87f-5df736ed2939","Type":"ContainerStarted","Data":"02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16"}
Nov 03 19:41:45.592874 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:45.592839    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 19:41:45.594877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:45.594782    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94" event={"ID":"752cf009-4d23-4820-ae58-b3ad21317e57","Type":"ContainerStarted","Data":"02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b"}
Nov 03 19:41:45.594877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:45.594844    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 19:41:45.596609 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:45.596577    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58" event={"ID":"40903bcb-9b64-4233-8230-5b74407797fb","Type":"ContainerStarted","Data":"07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313"}
Nov 03 19:41:45.596753 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:45.596635    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 19:41:45.610619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:45.610554    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m" podStartSLOduration=1.743859973 podStartE2EDuration="7.610536123s" totalImagesPullingTime="5.86667615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:45.608460813 +0000 UTC m=+618.878584109" watchObservedRunningTime="2026-11-03 19:41:45.610536123 +0000 UTC m=+618.880659422"
Nov 03 19:41:45.624608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:45.624557    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94" podStartSLOduration=1.686804151 podStartE2EDuration="7.624539119s" totalImagesPullingTime="5.937734968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:45.623425443 +0000 UTC m=+618.893548737" watchObservedRunningTime="2026-11-03 19:41:45.624539119 +0000 UTC m=+618.894662468"
Nov 03 19:41:45.642939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:45.642876    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58" podStartSLOduration=1.123659803 podStartE2EDuration="6.64285757s" totalImagesPullingTime="5.519197767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:45.639960245 +0000 UTC m=+618.910083541" watchObservedRunningTime="2026-11-03 19:41:45.64285757 +0000 UTC m=+618.912980868"
Nov 03 19:41:45.657832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:45.657800    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:45.798837 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:45.798700    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:46.671064 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:46.671032    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:47.608998 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:47.608955    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz" event={"ID":"dea0c8fc-05ab-4173-8084-ad7e2bc678ad","Type":"ContainerStarted","Data":"2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221"}
Nov 03 19:41:47.609603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:47.609199    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:47.628640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:47.628587    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz" podStartSLOduration=2.2935475419999998 podStartE2EDuration="5.62857242s" totalImagesPullingTime="3.335024878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:47.626420331 +0000 UTC m=+620.896543626" watchObservedRunningTime="2026-11-03 19:41:47.62857242 +0000 UTC m=+620.898695715"
Nov 03 19:41:47.670193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:47.670011    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:48.524218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:48.521671    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:49.225281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:49.225239    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:50.107950 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:50.107903    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:50.701950 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:50.701902    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:50.725505 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:41:50.725466    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:41:51.005254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:51.004869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:51.258252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:51.258121    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:51.417196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:51.417051    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:51.593315 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:51.584821    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:52.298452 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:52.260021    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:53.516449 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:53.516409    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:53.622387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:53.622359    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 19:41:54.771541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:54.771409    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:55.534499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:55.534399    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:55.601075 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:55.601043    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 19:41:55.676213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:55.676140    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:55.952448 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:55.952330    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:56.125519 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:56.125481    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:56.258422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:56.258303    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:56.401674 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:56.401641    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:56.549575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:56.549493    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:56.613877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:56.612825    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 19:41:56.629049 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:56.628088    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 19:41:56.629049 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:56.628421    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 19:41:56.629492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:56.629325    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 19:41:56.630475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:56.629929    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 19:41:56.631830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:56.630842    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 19:41:56.747317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:56.747218    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:57.088795 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:57.088751    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:57.219987 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:57.219913    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:57.654920 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:57.654828    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:58.784658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.784624    4531 generic.go:356] "Generic (PLEG): container finished" podID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerID="9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004" exitCode=0
Nov 03 19:41:58.785228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.784698    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" event={"ID":"e68ef5cf-00e6-423c-92f6-ff8200f459b1","Type":"ContainerDied","Data":"9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004"}
Nov 03 19:41:58.785228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.785210    4531 scope.go:122] "RemoveContainer" containerID="9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004"
Nov 03 19:41:58.786615 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.786595    4531 generic.go:356] "Generic (PLEG): container finished" podID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerID="89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547" exitCode=0
Nov 03 19:41:58.786718 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.786624    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" event={"ID":"fcdaf9c8-7009-4378-8f63-3071e17f5525","Type":"ContainerDied","Data":"89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547"}
Nov 03 19:41:58.786768 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.786733    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:58.786836 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.786819    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:58.786883 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.786868    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:58.786964 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.786926    4531 scope.go:122] "RemoveContainer" containerID="89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547"
Nov 03 19:41:58.788291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.788262    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:58.788386 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.788326    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:58.788386 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:58.788359    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:59.103310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.103219    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:59.795034 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.794995    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" event={"ID":"e68ef5cf-00e6-423c-92f6-ff8200f459b1","Type":"ContainerStarted","Data":"b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999"}
Nov 03 19:41:59.796904 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.796880    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" event={"ID":"fcdaf9c8-7009-4378-8f63-3071e17f5525","Type":"ContainerStarted","Data":"f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee"}
Nov 03 19:41:59.798352 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.798331    4531 generic.go:356] "Generic (PLEG): container finished" podID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerID="8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4" exitCode=0
Nov 03 19:41:59.798433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.798365    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" event={"ID":"aa2ffbba-e334-4eb9-b86b-184fc8c04bed","Type":"ContainerDied","Data":"8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4"}
Nov 03 19:41:59.798653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.798639    4531 scope.go:122] "RemoveContainer" containerID="8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4"
Nov 03 19:41:59.799978 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.799954    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:59.800031 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.800016    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:59.800066 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.800046    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:59.830234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.830212    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:41:59.830438 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.830420    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:41:59.830481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:41:59.830461    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 19:42:00.091267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:00.090868    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:00.387699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:00.382535    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:42:00.387699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:00.382677    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 19:42:00.387699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:00.382722    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 19:42:00.698909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:00.698778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:00.877950 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:00.877911    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:42:00.879815 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:00.879781    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 19:42:00.995293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:00.978955    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:01.126137 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.125766    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" event={"ID":"aa2ffbba-e334-4eb9-b86b-184fc8c04bed","Type":"ContainerStarted","Data":"ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d"}
Nov 03 19:42:01.234945 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.234897    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:01.378185 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.378091    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:01.382803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.382770    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 19:42:01.383058 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.383030    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 19:42:01.547198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.543882    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:01.655235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.651321    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:42:01.655235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.653014    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 19:42:01.655235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.653084    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 19:42:01.721861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.721821    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:01.831696 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.831664    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:42:01.831996 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.831969    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 19:42:01.868210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:01.866730    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:02.384198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:02.383618    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 19:42:02.384198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:02.383683    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 19:42:02.715252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:02.710645    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:02.715252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:02.711271    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 19:42:02.715252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:02.711334    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 19:42:02.832199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:02.830503    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:42:02.832199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:02.830565    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 19:42:03.368196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.367873    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:03.385198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.383685    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 19:42:03.385198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.383761    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 19:42:03.628368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.628264    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"]
Nov 03 19:42:03.645227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.632334    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.669225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.650617    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-39043d\"/\"kube-root-ca.crt\""
Nov 03 19:42:03.669225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.650677    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-39043d\"/\"openshift-service-ca.crt\""
Nov 03 19:42:03.669225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.650832    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39043d\"/\"simulator-w3-39043d-kserve-self-signed-certs\""
Nov 03 19:42:03.669225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.650939    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39043d\"/\"default-dockercfg-df4p9\""
Nov 03 19:42:03.669225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.651254    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 19:42:03.669225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.651301    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 19:42:03.669225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.656262    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"]
Nov 03 19:42:03.717199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.695077    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:03.732654 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.732622    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw558\" (UniqueName: \"kubernetes.io/projected/4bca7c22-ce45-445e-ba29-c522e440abe9-kube-api-access-lw558\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.732949 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.732926    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-tmp-dir\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.733118 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.733103    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-dshm\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.733261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.733246    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bca7c22-ce45-445e-ba29-c522e440abe9-tls-certs\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.733395 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.733382    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-model-cache\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.733625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.733609    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-home\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.846120 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.845619    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:42:03.846120 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.845700    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 19:42:03.896926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.873329    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-home\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.896926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.873402    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw558\" (UniqueName: \"kubernetes.io/projected/4bca7c22-ce45-445e-ba29-c522e440abe9-kube-api-access-lw558\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.896926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.873487    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-tmp-dir\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.896926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.873561    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-dshm\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.896926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.873603    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bca7c22-ce45-445e-ba29-c522e440abe9-tls-certs\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.896926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.873648    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-model-cache\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.896926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.874143    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-model-cache\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.896926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.893791    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-tmp-dir\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.896926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.894060    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-home\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.931197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.930617    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bca7c22-ce45-445e-ba29-c522e440abe9-tls-certs\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.931197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.930919    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-dshm\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:03.932995 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:03.932920    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw558\" (UniqueName: \"kubernetes.io/projected/4bca7c22-ce45-445e-ba29-c522e440abe9-kube-api-access-lw558\") pod \"simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") " pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:04.056198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.055400    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:04.193253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.171366    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:04.389959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.388629    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 19:42:04.389959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.388691    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 19:42:04.389959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.389120    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:42:04.628058 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.550994    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"]
Nov 03 19:42:04.635999 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.635968    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"]
Nov 03 19:42:04.636334 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.636288    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.698290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.645677    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0486e8\"/\"default-dockercfg-s9t9z\""
Nov 03 19:42:04.698290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.646135    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0486e8\"/\"openshift-service-ca.crt\""
Nov 03 19:42:04.698290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.646240    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0486e8\"/\"simulator-w5-0486e8-kserve-self-signed-certs\""
Nov 03 19:42:04.698290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.646309    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0486e8\"/\"kube-root-ca.crt\""
Nov 03 19:42:04.698290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.688552    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"]
Nov 03 19:42:04.731415 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.706786    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 19:42:04.731415 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.706853    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 19:42:04.731415 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.710726    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:04.733419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.732895    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-tmp-dir\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.733419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.733042    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-model-cache\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.733419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.733069    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bc662c-d375-4d6a-83eb-32481c8f7902-tls-certs\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.733419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.733131    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-home\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.733419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.733242    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-dshm\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.733419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.733268    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nqpp\" (UniqueName: \"kubernetes.io/projected/d4bc662c-d375-4d6a-83eb-32481c8f7902-kube-api-access-9nqpp\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.840203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.831559    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:42:04.840203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.831620    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 03 19:42:04.866200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.853505    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-home\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.891679 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.891643    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-dshm\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.891935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.891912    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqpp\" (UniqueName: \"kubernetes.io/projected/d4bc662c-d375-4d6a-83eb-32481c8f7902-kube-api-access-9nqpp\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.892083 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.892068    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-tmp-dir\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.913284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.913254    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-model-cache\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.917260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.917227    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bc662c-d375-4d6a-83eb-32481c8f7902-tls-certs\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.941408 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.866838    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-home\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.941408 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.919220    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-model-cache\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.941408 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:04.912364    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-tmp-dir\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:04.941408 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:42:04.875455    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bca7c22_ce45_445e_ba29_c522e440abe9.slice/crio-c0918ce1ac97d62d00a481e84c08cce233ce9287adbe1143a68a5869958fc1b5 WatchSource:0}: Error finding container c0918ce1ac97d62d00a481e84c08cce233ce9287adbe1143a68a5869958fc1b5: Status 404 returned error can't find the container with id c0918ce1ac97d62d00a481e84c08cce233ce9287adbe1143a68a5869958fc1b5
Nov 03 19:42:05.011831 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.011780    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bc662c-d375-4d6a-83eb-32481c8f7902-tls-certs\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:05.045288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.039739    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-dshm\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:05.045288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.040239    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:05.208406 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.206901    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nqpp\" (UniqueName: \"kubernetes.io/projected/d4bc662c-d375-4d6a-83eb-32481c8f7902-kube-api-access-9nqpp\") pod \"simulator-w5-0486e8-kserve-798bd88bbf-kf4ml\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") " pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:05.215339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.215219    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" event={"ID":"4bca7c22-ce45-445e-ba29-c522e440abe9","Type":"ContainerStarted","Data":"c0918ce1ac97d62d00a481e84c08cce233ce9287adbe1143a68a5869958fc1b5"}
Nov 03 19:42:05.394407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.394363    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:42:05.394878 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.394794    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:05.395561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.395322    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:42:05.395561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.395510    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 19:42:05.395758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.395568    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 19:42:05.668246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.665823    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 19:42:05.668246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.665886    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 19:42:05.757426 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.757350    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"]
Nov 03 19:42:05.759092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.759009    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:05.930933 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:05.930383    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:42:06.074692 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.074577    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:06.223585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.223546    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" event={"ID":"4bca7c22-ce45-445e-ba29-c522e440abe9","Type":"ContainerStarted","Data":"2fd6a943663d357d6788d3f1af7af3531e66d9477f007e5c13cbd6a2b324a3d4"}
Nov 03 19:42:06.225164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.225135    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" event={"ID":"d4bc662c-d375-4d6a-83eb-32481c8f7902","Type":"ContainerStarted","Data":"1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0"}
Nov 03 19:42:06.225288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.225190    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" event={"ID":"d4bc662c-d375-4d6a-83eb-32481c8f7902","Type":"ContainerStarted","Data":"6c9e9d9cb95a0064509833ca8ba04e96736bc60fff949f89fd8d6c6cdc0f8929"}
Nov 03 19:42:06.226532 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.226511    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 19:42:06.253197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.253042    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podStartSLOduration=3.253022126 podStartE2EDuration="3.253022126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:06.250600793 +0000 UTC m=+639.520724089" watchObservedRunningTime="2026-11-03 19:42:06.253022126 +0000 UTC m=+639.523145422"
Nov 03 19:42:06.270375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.270311    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podStartSLOduration=2.270289134 podStartE2EDuration="2.270289134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:06.266798286 +0000 UTC m=+639.536921581" watchObservedRunningTime="2026-11-03 19:42:06.270289134 +0000 UTC m=+639.540412433"
Nov 03 19:42:06.385950 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.385921    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:42:06.387013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.386992    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 19:42:06.395349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.395322    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:06.544940 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.544901    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:06.688900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.688809    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:06.781400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.781367    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:42:06.783434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.783405    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 19:42:06.850543 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.850511    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:06.995717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:06.995635    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:07.056300 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.056270    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:07.127051 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.127015    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"]
Nov 03 19:42:07.133976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.133945    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.139684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.139656    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"]
Nov 03 19:42:07.139880 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.139694    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"openshift-service-ca.crt\""
Nov 03 19:42:07.140019 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.139941    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"kube-root-ca.crt\""
Nov 03 19:42:07.140231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.140211    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"simulator-w6-102379-kserve-self-signed-certs\""
Nov 03 19:42:07.140329 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.140305    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"default-dockercfg-cv6dk\""
Nov 03 19:42:07.233519 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.233490    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:07.328695 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.328606    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-tmp-dir\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.328849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.328812    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-model-cache\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.328944 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.328848    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4699bd7-90b4-477c-bc1f-c819a5675de2-tls-certs\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.329506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.329477    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-dshm\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.329756 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.329725    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-home\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.329865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.329811    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5vw\" (UniqueName: \"kubernetes.io/projected/a4699bd7-90b4-477c-bc1f-c819a5675de2-kube-api-access-4c5vw\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.434240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.432823    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-home\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.434240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.432892    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5vw\" (UniqueName: \"kubernetes.io/projected/a4699bd7-90b4-477c-bc1f-c819a5675de2-kube-api-access-4c5vw\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.434240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.432964    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-tmp-dir\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.434240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.433018    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-model-cache\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.434240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.433043    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4699bd7-90b4-477c-bc1f-c819a5675de2-tls-certs\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.434240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.433190    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-dshm\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.434240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.433754    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-model-cache\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.434240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.434002    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-home\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.435094 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.434895    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-tmp-dir\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.437051 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.437027    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4699bd7-90b4-477c-bc1f-c819a5675de2-tls-certs\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.444848 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.444780    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5vw\" (UniqueName: \"kubernetes.io/projected/a4699bd7-90b4-477c-bc1f-c819a5675de2-kube-api-access-4c5vw\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.454139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.454102    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-dshm\") pod \"simulator-w6-102379-kserve-8cd95d64c-m67zs\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") " pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:07.595273 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.588849    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:07.753789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:07.753754    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:08.058828 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:08.058797    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"]
Nov 03 19:42:08.269195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:08.265832    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" event={"ID":"a4699bd7-90b4-477c-bc1f-c819a5675de2","Type":"ContainerStarted","Data":"992d93fc660d58139c7add18026e6b68dcc7111876570f7f63b41022517fdcad"}
Nov 03 19:42:08.269195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:08.265884    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" event={"ID":"a4699bd7-90b4-477c-bc1f-c819a5675de2","Type":"ContainerStarted","Data":"5469a9d834cd8e15ad968c970a27b4a29e994eadc32b8e31f9cf1aa21f277d17"}
Nov 03 19:42:08.287397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:08.287327    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podStartSLOduration=1.287311231 podStartE2EDuration="1.287311231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:08.284321612 +0000 UTC m=+641.554444912" watchObservedRunningTime="2026-11-03 19:42:08.287311231 +0000 UTC m=+641.557434527"
Nov 03 19:42:08.652812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:08.652764    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:08.754064 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:08.754018    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:09.340432 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:09.340379    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:10.052575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:10.052536    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:10.212615 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:10.208632    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:10.716696 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:10.705297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:11.435202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.430368    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:11.919702 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.919670    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"]
Nov 03 19:42:11.922650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.922632    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:11.929714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.929667    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f4e123\"/\"openshift-service-ca.crt\""
Nov 03 19:42:11.929714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.929683    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f4e123\"/\"kube-root-ca.crt\""
Nov 03 19:42:11.930401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.930373    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f4e123\"/\"default-dockercfg-dzh8r\""
Nov 03 19:42:11.930545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.930428    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f4e123\"/\"simulator-w4-f4e123-kserve-self-signed-certs\""
Nov 03 19:42:11.933216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.933194    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"]
Nov 03 19:42:11.988299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.988268    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-tmp-dir\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:11.988299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.988303    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-home\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:11.988539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.988428    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5effd563-1417-489b-bbc4-74a93b08d24d-tls-certs\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:11.988539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.988488    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-dshm\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:11.988645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.988615    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-model-cache\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:11.988702 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:11.988659    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxhz\" (UniqueName: \"kubernetes.io/projected/5effd563-1417-489b-bbc4-74a93b08d24d-kube-api-access-kzxhz\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.004299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.004269    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:12.094197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.090419    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5effd563-1417-489b-bbc4-74a93b08d24d-tls-certs\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.094197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.090532    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-dshm\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.094197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.090585    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-model-cache\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.094197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.090616    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxhz\" (UniqueName: \"kubernetes.io/projected/5effd563-1417-489b-bbc4-74a93b08d24d-kube-api-access-kzxhz\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.094197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.090665    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-tmp-dir\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.094197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.090689    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-home\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.094197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.091634    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-model-cache\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.094714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.094598    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-tmp-dir\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.100389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.097204    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-home\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.116090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.115541    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5effd563-1417-489b-bbc4-74a93b08d24d-tls-certs\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.120150 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.120104    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxhz\" (UniqueName: \"kubernetes.io/projected/5effd563-1417-489b-bbc4-74a93b08d24d-kube-api-access-kzxhz\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.121039 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.120979    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-dshm\") pod \"simulator-w4-f4e123-kserve-55b9dc685d-q942q\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") " pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.137757 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.137726    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:12.236433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.236348    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:12.508519 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.508339    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"]
Nov 03 19:42:12.511271 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:42:12.511226    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5effd563_1417_489b_bbc4_74a93b08d24d.slice/crio-eba440639f2246bc068b5a7dc2ace4642d92513f0865937d8ffeb5ef6c88ff98 WatchSource:0}: Error finding container eba440639f2246bc068b5a7dc2ace4642d92513f0865937d8ffeb5ef6c88ff98: Status 404 returned error can't find the container with id eba440639f2246bc068b5a7dc2ace4642d92513f0865937d8ffeb5ef6c88ff98
Nov 03 19:42:12.816085 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:12.812826    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:13.293664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:13.293623    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" event={"ID":"5effd563-1417-489b-bbc4-74a93b08d24d","Type":"ContainerStarted","Data":"2dca25c01e61f165c79beca85476e3d37c1a967f15be6a27a396bf21327e6959"}
Nov 03 19:42:13.293664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:13.293676    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" event={"ID":"5effd563-1417-489b-bbc4-74a93b08d24d","Type":"ContainerStarted","Data":"eba440639f2246bc068b5a7dc2ace4642d92513f0865937d8ffeb5ef6c88ff98"}
Nov 03 19:42:13.318912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:13.318855    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podStartSLOduration=2.318838494 podStartE2EDuration="2.318838494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:13.316723444 +0000 UTC m=+646.586846739" watchObservedRunningTime="2026-11-03 19:42:13.318838494 +0000 UTC m=+646.588961789"
Nov 03 19:42:13.508917 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:13.508875    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:14.056857 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:14.056814    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:14.069705 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:14.069676    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:14.133229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:14.133192    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:14.236957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:14.236923    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:14.310615 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:14.310531    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:42:14.906109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:14.903623    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:15.142682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:15.142647    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:15.275697 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:15.275600    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:15.395625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:15.395587    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:15.408536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:15.408507    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:15.831203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:15.830344    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:16.323475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:16.323441    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:42:16.544503 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:16.544467    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:17.254781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:17.254679    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:17.754265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:17.754232    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:17.767321 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:17.767288    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:17.947198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:17.945031    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:18.087729 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.087640    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"]
Nov 03 19:42:18.090639 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.090610    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.097793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.097469    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-82dbbf\"/\"openshift-service-ca.crt\""
Nov 03 19:42:18.097967 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.097873    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-82dbbf\"/\"simulator-w0-82dbbf-kserve-self-signed-certs\""
Nov 03 19:42:18.097967 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.097951    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-82dbbf\"/\"default-dockercfg-lc8tj\""
Nov 03 19:42:18.097967 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.097953    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-82dbbf\"/\"kube-root-ca.crt\""
Nov 03 19:42:18.101927 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.101899    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"]
Nov 03 19:42:18.173754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.173714    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:18.178043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.178012    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-model-cache\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.178214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.178057    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44825f51-d618-4095-93fe-478a2f497eeb-tls-certs\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.178214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.178084    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnpjm\" (UniqueName: \"kubernetes.io/projected/44825f51-d618-4095-93fe-478a2f497eeb-kube-api-access-fnpjm\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.178362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.178270    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-dshm\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.178398 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.178360    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-home\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.178448 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.178412    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-tmp-dir\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.280611 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.280560    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-model-cache\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.280798 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.280635    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44825f51-d618-4095-93fe-478a2f497eeb-tls-certs\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.280798 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.280666    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnpjm\" (UniqueName: \"kubernetes.io/projected/44825f51-d618-4095-93fe-478a2f497eeb-kube-api-access-fnpjm\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.280798 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.280735    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-dshm\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.280998 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.280802    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-home\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.280998 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.280863    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-tmp-dir\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.281857 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.281811    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-tmp-dir\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.282327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.282294    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-home\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.282587 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.282510    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-model-cache\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.287441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.287408    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-dshm\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.289021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.288987    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44825f51-d618-4095-93fe-478a2f497eeb-tls-certs\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.289851 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.289826    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnpjm\" (UniqueName: \"kubernetes.io/projected/44825f51-d618-4095-93fe-478a2f497eeb-kube-api-access-fnpjm\") pod \"simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") " pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.333467 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.333436    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:42:18.405639 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.405600    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:18.635104 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.635046    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:18.681489 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:18.681460    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"]
Nov 03 19:42:18.683761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:42:18.683717    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44825f51_d618_4095_93fe_478a2f497eeb.slice/crio-031f46f2cddb1dd369ba8d1fb22fd7d0c1c175912012ee473bf3feaebd3de830 WatchSource:0}: Error finding container 031f46f2cddb1dd369ba8d1fb22fd7d0c1c175912012ee473bf3feaebd3de830: Status 404 returned error can't find the container with id 031f46f2cddb1dd369ba8d1fb22fd7d0c1c175912012ee473bf3feaebd3de830
Nov 03 19:42:19.246531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:19.246473    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:19.329408 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:19.329364    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" event={"ID":"44825f51-d618-4095-93fe-478a2f497eeb","Type":"ContainerStarted","Data":"0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1"}
Nov 03 19:42:19.329557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:19.329421    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" event={"ID":"44825f51-d618-4095-93fe-478a2f497eeb","Type":"ContainerStarted","Data":"031f46f2cddb1dd369ba8d1fb22fd7d0c1c175912012ee473bf3feaebd3de830"}
Nov 03 19:42:19.344957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:19.344904    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podStartSLOduration=1.344891159 podStartE2EDuration="1.344891159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:19.343070062 +0000 UTC m=+652.613193356" watchObservedRunningTime="2026-11-03 19:42:19.344891159 +0000 UTC m=+652.615014454"
Nov 03 19:42:19.405761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:19.405728    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:19.869869 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:19.866713    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:20.533063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:20.531419    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:20.573551 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:42:20.573496    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:42:21.252514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:21.252476    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:22.146098 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:22.146054    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:22.236544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:22.236502    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:22.255881 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:22.255848    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:22.279443 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:22.279414    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:22.356677 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:22.356653    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:42:22.831493 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:22.831461    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:23.472345 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:23.472260    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:24.070593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:24.070562    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:24.199537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:24.199506    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:24.848914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:24.848773    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:25.641199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:25.641122    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:26.419541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:26.419500    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:27.154206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:27.154155    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:27.822651 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:27.822545    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:28.406108 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:28.406075    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:28.418698 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:28.418664    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:28.654443 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:28.654406    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:29.402299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:29.402269    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:42:29.510056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:29.510023    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:30.233635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.233600    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:30.318009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.317963    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"]
Nov 03 19:42:30.326031 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.326004    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.326607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.326056    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"]
Nov 03 19:42:30.333964 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.333928    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"openshift-service-ca.crt\""
Nov 03 19:42:30.334209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.334161    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"kube-root-ca.crt\""
Nov 03 19:42:30.334761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.334463    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"premium-w1-a35658-kserve-self-signed-certs\""
Nov 03 19:42:30.334761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.334566    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"default-dockercfg-s6mct\""
Nov 03 19:42:30.397542 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.397508    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:30.465197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.447456    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tls-certs\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.465197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.447655    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tmp-dir\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.465197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.447734    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-dshm\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.465197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.447770    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-model-cache\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.465197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.447874    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v955\" (UniqueName: \"kubernetes.io/projected/d89c40ad-f434-42e9-a9b3-5c7311c380bd-kube-api-access-4v955\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.465197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.448002    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-home\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.548685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.548599    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tmp-dir\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.548685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.548656    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-dshm\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.548685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.548680    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-model-cache\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.548977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.548740    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v955\" (UniqueName: \"kubernetes.io/projected/d89c40ad-f434-42e9-a9b3-5c7311c380bd-kube-api-access-4v955\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.548977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.548798    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-home\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.548977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.548831    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tls-certs\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.549142 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.549030    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tmp-dir\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.549142 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.549075    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-model-cache\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.549264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.549149    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-home\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.551098 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.551074    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-dshm\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.551407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.551387    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tls-certs\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.559360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.559339    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v955\" (UniqueName: \"kubernetes.io/projected/d89c40ad-f434-42e9-a9b3-5c7311c380bd-kube-api-access-4v955\") pod \"premium-w1-a35658-kserve-6646c4894c-lj7dm\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") " pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.641557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.641518    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:30.919653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:30.919607    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"]
Nov 03 19:42:30.929533 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:42:30.929495    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd89c40ad_f434_42e9_a9b3_5c7311c380bd.slice/crio-70c8ba191ec242ad9625c8229f428e312a3d81f673473aabcdbc223c8ba6a587 WatchSource:0}: Error finding container 70c8ba191ec242ad9625c8229f428e312a3d81f673473aabcdbc223c8ba6a587: Status 404 returned error can't find the container with id 70c8ba191ec242ad9625c8229f428e312a3d81f673473aabcdbc223c8ba6a587
Nov 03 19:42:31.031322 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:31.031281    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:31.398253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:31.398208    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" event={"ID":"d89c40ad-f434-42e9-a9b3-5c7311c380bd","Type":"ContainerStarted","Data":"c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d"}
Nov 03 19:42:31.398253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:31.398259    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" event={"ID":"d89c40ad-f434-42e9-a9b3-5c7311c380bd","Type":"ContainerStarted","Data":"70c8ba191ec242ad9625c8229f428e312a3d81f673473aabcdbc223c8ba6a587"}
Nov 03 19:42:31.415510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:31.415453    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podStartSLOduration=1.415439971 podStartE2EDuration="1.415439971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:31.414933591 +0000 UTC m=+664.685056889" watchObservedRunningTime="2026-11-03 19:42:31.415439971 +0000 UTC m=+664.685563266"
Nov 03 19:42:31.642209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:31.642157    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:31.846830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:31.846743    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:32.590538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:32.588076    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:33.081052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:33.081017    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:33.356257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:33.352446    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:34.044219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:34.040775    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:34.736203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:34.736141    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:34.888873 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:34.888839    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:35.422509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:35.422472    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:36.146613 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:36.146575    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:36.827704 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:36.822563    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:37.473212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.473162    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"]
Nov 03 19:42:37.477928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.477905    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.484400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.484356    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0486e8\"/\"premium-w5-0486e8-kserve-self-signed-certs\""
Nov 03 19:42:37.487055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.487030    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"]
Nov 03 19:42:37.531482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.531446    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:37.650002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.649732    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60b85d3-ab1a-482a-86f0-32ce873c7383-tls-certs\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.650002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.649819    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-dshm\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.650002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.649876    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gqkp\" (UniqueName: \"kubernetes.io/projected/c60b85d3-ab1a-482a-86f0-32ce873c7383-kube-api-access-4gqkp\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.650002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.649948    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-tmp-dir\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.650002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.649993    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-home\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.650472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.650111    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-model-cache\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.661190 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.661103    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:37.751327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.751236    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-home\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.751327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.751322    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-model-cache\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.751567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.751357    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60b85d3-ab1a-482a-86f0-32ce873c7383-tls-certs\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.751567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.751393    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-dshm\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.751567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.751426    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gqkp\" (UniqueName: \"kubernetes.io/projected/c60b85d3-ab1a-482a-86f0-32ce873c7383-kube-api-access-4gqkp\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.751567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.751472    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-tmp-dir\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.751789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.751682    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-home\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.751789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.751759    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-model-cache\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.751902 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.751881    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-tmp-dir\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.753701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.753677    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-dshm\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.753976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.753961    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60b85d3-ab1a-482a-86f0-32ce873c7383-tls-certs\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.759400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.759376    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gqkp\" (UniqueName: \"kubernetes.io/projected/c60b85d3-ab1a-482a-86f0-32ce873c7383-kube-api-access-4gqkp\") pod \"premium-w5-0486e8-kserve-576798c84d-fnhwk\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") " pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:37.794251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:37.794211    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:38.068972 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:38.068943    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"]
Nov 03 19:42:38.070645 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:42:38.070612    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc60b85d3_ab1a_482a_86f0_32ce873c7383.slice/crio-fe2235e494baa38af00832f5e2a1a7d2600f2e01c78a230fd72fe5fa0e48412f WatchSource:0}: Error finding container fe2235e494baa38af00832f5e2a1a7d2600f2e01c78a230fd72fe5fa0e48412f: Status 404 returned error can't find the container with id fe2235e494baa38af00832f5e2a1a7d2600f2e01c78a230fd72fe5fa0e48412f
Nov 03 19:42:38.228807 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:38.228726    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:38.437523 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:38.437484    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" event={"ID":"c60b85d3-ab1a-482a-86f0-32ce873c7383","Type":"ContainerStarted","Data":"0850ac68e9b5439e0ffef86e46c46ae13c412829c02ca3a9728d47abb1d8cfce"}
Nov 03 19:42:38.437712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:38.437532    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" event={"ID":"c60b85d3-ab1a-482a-86f0-32ce873c7383","Type":"ContainerStarted","Data":"fe2235e494baa38af00832f5e2a1a7d2600f2e01c78a230fd72fe5fa0e48412f"}
Nov 03 19:42:38.454439 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:38.454384    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podStartSLOduration=1.4543701709999999 podStartE2EDuration="1.454370171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:38.451563071 +0000 UTC m=+671.721686367" watchObservedRunningTime="2026-11-03 19:42:38.454370171 +0000 UTC m=+671.724493465"
Nov 03 19:42:38.795393 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:38.795311    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:38.988881 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:38.988840    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:39.749107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:39.744297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:40.392554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:40.392501    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:40.521517 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:40.521477    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:40.641783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:40.641748    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:40.654544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:40.654472    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:41.128714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:41.128683    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:41.471866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:41.471789    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:42:41.761609 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:41.761499    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:42.410385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:42.410349    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:43.138741 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:43.138705    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:43.747253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:43.747203    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:44.506925 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:44.506886    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:45.553897 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:45.553858    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:46.080319 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:46.080288    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:46.203656 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:46.203622    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:46.329788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:46.329756    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:47.182712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:47.182674    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:47.651238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:47.641361    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:47.794581 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:47.794547    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:47.806949 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:47.806924    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:48.438273 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:48.438217    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:48.509602 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:48.509568    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:42:49.149966 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:49.149916    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:49.495337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:49.495259    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:49.958048 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:49.958013    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:50.575193 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:42:50.575141    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:42:50.834131 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:50.834047    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:51.532082 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:51.532025    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:52.205637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:52.205606    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:52.860132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:52.859824    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:53.534216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:53.528296    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:54.207844 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:54.207806    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:54.927598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:54.927567    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:56.032348 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:56.032309    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:57.050771 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:57.050721    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:57.821084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:57.820058    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:58.727126 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:58.727091    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:59.448204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:42:59.444755    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:00.234084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:00.234046    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:01.040902 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:01.040869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:01.646222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:01.645334    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:02.289015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.288980    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"]
Nov 03 19:43:02.294378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.294350    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.301314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.301282    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"]
Nov 03 19:43:02.302638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.302599    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-39043d-openshift-default-dockercfg-zwlqs\""
Nov 03 19:43:02.355733 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.341909    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:02.368075 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.368031    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"]
Nov 03 19:43:02.372465 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.372442    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.382761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.382729    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"]
Nov 03 19:43:02.393483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.393453    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.393630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.393493    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.393630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.393519    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.393630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.393554    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjbfp\" (UniqueName: \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-kube-api-access-zjbfp\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.393630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.393597    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.393832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.393633    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.393832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.393721    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.393832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.393745    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.393832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.393779    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.400739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.400713    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a35658-openshift-default-dockercfg-m75kt\""
Nov 03 19:43:02.494852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.494812    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.494852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.494854    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.495063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.494889    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.495063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.494982    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjbfp\" (UniqueName: \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-kube-api-access-zjbfp\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.495063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495030    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.495190 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495071    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.495190 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495114    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.495271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495215    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.495307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495282    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.495346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495325    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.495378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495355    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.495423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495398    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.495464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495429    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.495516 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495493    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvq52\" (UniqueName: \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-kube-api-access-cvq52\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.495572 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495560    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.495630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495574    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.495630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495610    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.495741 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495677    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.495741 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495700    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.495846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495736    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.495846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.495807    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.496054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.496038    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.496275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.496252    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.497702 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.497680    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.497836 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.497820    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.501848 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.501827    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.502088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.502058    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjbfp\" (UniqueName: \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-kube-api-access-zjbfp\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.596957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.596873    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.596957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.596926    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.596957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.596953    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.597255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.596982    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvq52\" (UniqueName: \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-kube-api-access-cvq52\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.597255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.597107    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.597255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.597157    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.597255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.597236    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.597467 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.597271    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.597467 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.597335    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.597575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.597514    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.597657 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.597629    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.597793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.597726    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.597793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.597761    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.597934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.597915    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.599501 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.599478    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.599643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.599623    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.607650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.607620    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvq52\" (UniqueName: \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-kube-api-access-cvq52\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.607774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.607625    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.620456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.620426    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:02.691764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.691456    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:02.840744 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.840713    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"]
Nov 03 19:43:02.845701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.845685    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:02.851920 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.851234    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-102379-openshift-default-dockercfg-2s5bm\""
Nov 03 19:43:02.862825 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.862803    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"]
Nov 03 19:43:02.912057 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.912029    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"]
Nov 03 19:43:02.913551 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:43:02.913525    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d62963_94d7_4cf8_a1e9_2b73bb046ca7.slice/crio-ac81d90c169754c79bd97ada88ce6f043ee759c3a2738b6b1b73bbd1f393c6f9 WatchSource:0}: Error finding container ac81d90c169754c79bd97ada88ce6f043ee759c3a2738b6b1b73bbd1f393c6f9: Status 404 returned error can't find the container with id ac81d90c169754c79bd97ada88ce6f043ee759c3a2738b6b1b73bbd1f393c6f9
Nov 03 19:43:02.916029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.915998    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:43:02.916133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.916076    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:43:02.916133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.916123    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:43:02.978460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.978434    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"]
Nov 03 19:43:02.979528 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:43:02.979504    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6a58ee_f52f_49ef_b0a7_d89dd10804d9.slice/crio-09e90efd9bca4b4ff3e4a4c045172b80ec276f697395f830e95920a3d275b66b WatchSource:0}: Error finding container 09e90efd9bca4b4ff3e4a4c045172b80ec276f697395f830e95920a3d275b66b: Status 404 returned error can't find the container with id 09e90efd9bca4b4ff3e4a4c045172b80ec276f697395f830e95920a3d275b66b
Nov 03 19:43:02.981931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.981898    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:43:02.982031 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.981969    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:43:02.982031 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:02.982003    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:43:03.001700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.001654    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb786573-56fb-4acd-b100-5ac76af38892-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.001871 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.001847    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clcgl\" (UniqueName: \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-kube-api-access-clcgl\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.002101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.002081    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.002198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.002115    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb786573-56fb-4acd-b100-5ac76af38892-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.002198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.002143    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.002334 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.002226    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.002334 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.002293    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.002436 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.002412    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.002488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.002459    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.104280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.104154    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb786573-56fb-4acd-b100-5ac76af38892-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.104280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.104241    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcgl\" (UniqueName: \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-kube-api-access-clcgl\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.104514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.104349    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.104514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.104377    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb786573-56fb-4acd-b100-5ac76af38892-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.104514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.104403    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.104514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.104427    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.104514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.104454    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.104771 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.104528    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.104771 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.104576    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.104973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.104944    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb786573-56fb-4acd-b100-5ac76af38892-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.105072 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.105023    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.105072 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.105040    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.105351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.105307    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.105724 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.105701    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.107273 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.107247    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.107723 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.107698    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb786573-56fb-4acd-b100-5ac76af38892-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.111291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.111260    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.111819 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.111801    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcgl\" (UniqueName: \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-kube-api-access-clcgl\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.145241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.133795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:03.164459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.164417    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:03.453823 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.453783    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"]
Nov 03 19:43:03.486310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.486271    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:43:03.487916 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.487879    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:43:03.488113 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.488093    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:43:03.594878 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.594647    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" event={"ID":"cb786573-56fb-4acd-b100-5ac76af38892","Type":"ContainerStarted","Data":"ff70392365b959e1fc7f7054e63c899ef53ef14463ced7e8d9ff44d01f50712e"}
Nov 03 19:43:03.596892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.596356    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" event={"ID":"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9","Type":"ContainerStarted","Data":"eaa4cdc6f096264231d1fffc411e03e12cd7d929eb47f989a1300a6d6b2ccf87"}
Nov 03 19:43:03.596892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.596397    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" event={"ID":"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9","Type":"ContainerStarted","Data":"09e90efd9bca4b4ff3e4a4c045172b80ec276f697395f830e95920a3d275b66b"}
Nov 03 19:43:03.598343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.598317    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" event={"ID":"23d62963-94d7-4cf8-a1e9-2b73bb046ca7","Type":"ContainerStarted","Data":"b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8"}
Nov 03 19:43:03.598441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.598364    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" event={"ID":"23d62963-94d7-4cf8-a1e9-2b73bb046ca7","Type":"ContainerStarted","Data":"ac81d90c169754c79bd97ada88ce6f043ee759c3a2738b6b1b73bbd1f393c6f9"}
Nov 03 19:43:03.613460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.613396    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" podStartSLOduration=1.613378712 podStartE2EDuration="1.613378712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:03.612598513 +0000 UTC m=+696.882721810" watchObservedRunningTime="2026-11-03 19:43:03.613378712 +0000 UTC m=+696.883502011"
Nov 03 19:43:03.620911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.620889    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:03.622444 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.622404    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 19:43:03.622541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.622467    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 19:43:03.631532 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.631484    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" podStartSLOduration=1.6314717349999999 podStartE2EDuration="1.631471735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:03.62747439 +0000 UTC m=+696.897597685" watchObservedRunningTime="2026-11-03 19:43:03.631471735 +0000 UTC m=+696.901595030"
Nov 03 19:43:03.692530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.692475    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:03.832854 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:03.832759    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:04.528144 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:04.526586    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:04.629197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:04.625553    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 19:43:04.629197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:04.625616    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 19:43:04.634242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:04.633617    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" event={"ID":"cb786573-56fb-4acd-b100-5ac76af38892","Type":"ContainerStarted","Data":"e9dd083a1b59d4f7975ff7ccaf73646ca402812d6ca9ebdb64b6d28d91157b33"}
Nov 03 19:43:04.700090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:04.700010    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podStartSLOduration=2.699986424 podStartE2EDuration="2.699986424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:04.669790627 +0000 UTC m=+697.939913922" watchObservedRunningTime="2026-11-03 19:43:04.699986424 +0000 UTC m=+697.970109723"
Nov 03 19:43:04.717223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:04.702671    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:43:04.717416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:04.717286    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" podUID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:43:05.175261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:05.175225    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:05.175671 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:05.175645    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:43:05.175845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:05.175821    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:43:05.225018 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:05.224978    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:05.621559 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:05.621462    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 19:43:05.621783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:05.621754    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 19:43:05.701199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:05.693609    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:43:05.701199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:05.693673    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" podUID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:43:06.151200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:06.148702    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:06.171197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:06.170735    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:43:06.171197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:06.170812    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:43:06.621837 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:06.621755    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 19:43:06.622042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:06.622013    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 19:43:06.709221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:06.697258    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:43:06.709221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:06.697331    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" podUID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:43:06.872248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:06.871575    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:07.168941 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:07.168902    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:43:07.172070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:07.172029    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:43:07.621255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:07.621150    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 19:43:07.621255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:07.621234    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 19:43:07.643789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:07.643705    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:07.692388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:07.691890    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:43:07.692388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:07.691957    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" podUID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:43:08.166685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:08.166644    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:43:08.167318 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:08.167283    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:43:08.511256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:08.508720    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:08.626957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:08.626921    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 19:43:08.627257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:08.627226    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 19:43:08.707588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:08.707553    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:43:08.707856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:08.707824    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" podUID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:43:09.169617 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:09.169574    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:43:09.170260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:09.170224    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:43:09.372905 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:09.372719    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:09.621932 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:09.621840    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 19:43:09.621932 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:09.621900    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 19:43:09.692198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:09.692139    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:43:09.692372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:09.692228    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" podUID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:43:10.171269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:10.165215    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:10.176354 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:10.176313    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:43:10.176664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:10.176629    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:43:10.626432 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:10.626335    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:10.626652 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:10.626631    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:10.627424 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:10.627400    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:43:10.703875 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:10.703838    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:10.704846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:10.704606    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:10.705979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:10.705952    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 19:43:10.852331 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:10.852300    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:11.170829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:11.170793    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:11.171102 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:11.171078    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:11.171720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:11.171702    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 19:43:11.685783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:11.685746    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:12.663453 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:12.663412    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:13.587926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:13.587886    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:14.726198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:14.726141    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:15.200120 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:15.200083    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:15.471963 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:15.471871    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:16.279228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:16.279185    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:16.979128 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:16.979083    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:17.765751 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:17.759267    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:18.645869 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:18.645833    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:19.476933 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:19.476885    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:20.352860 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:20.352811    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:20.573329 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:43:20.573296    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:43:21.169390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:21.169353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:21.962971 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:21.962936    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:22.789086 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:22.789030    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:23.564502 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.564465    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:23.742907 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.742869    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"]
Nov 03 19:43:23.748897 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.748870    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:23.752448 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.752411    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:23.752448 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.752433    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:43:23.753405 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.753382    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"]
Nov 03 19:43:23.876774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.876690    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrfd\" (UniqueName: \"kubernetes.io/projected/c75ded06-8e99-4a28-be0b-8de4717a6999-kube-api-access-qbrfd\") pod \"maas-api-revoke-sub-7c684deb-fwnvz\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:23.876774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.876729    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c684deb-fwnvz\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:23.876774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.876759    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c684deb-fwnvz\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:23.977835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.977799    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrfd\" (UniqueName: \"kubernetes.io/projected/c75ded06-8e99-4a28-be0b-8de4717a6999-kube-api-access-qbrfd\") pod \"maas-api-revoke-sub-7c684deb-fwnvz\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:23.977835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.977837    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c684deb-fwnvz\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:23.978013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.977882    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c684deb-fwnvz\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:23.978544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.978522    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c684deb-fwnvz\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:23.980375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.980351    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c684deb-fwnvz\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:23.986358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:23.986335    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrfd\" (UniqueName: \"kubernetes.io/projected/c75ded06-8e99-4a28-be0b-8de4717a6999-kube-api-access-qbrfd\") pod \"maas-api-revoke-sub-7c684deb-fwnvz\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:24.065318 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:24.065289    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:24.353041 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:24.353004    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:24.362044 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:43:24.361998    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75ded06_8e99_4a28_be0b_8de4717a6999.slice/crio-178eeb009b60d3efaea355a45b662eb0de1a104550892b8fc7905beea7e7c43a WatchSource:0}: Error finding container 178eeb009b60d3efaea355a45b662eb0de1a104550892b8fc7905beea7e7c43a: Status 404 returned error can't find the container with id 178eeb009b60d3efaea355a45b662eb0de1a104550892b8fc7905beea7e7c43a
Nov 03 19:43:24.367411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:24.367380    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"]
Nov 03 19:43:24.839217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:24.839164    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz" event={"ID":"c75ded06-8e99-4a28-be0b-8de4717a6999","Type":"ContainerStarted","Data":"178eeb009b60d3efaea355a45b662eb0de1a104550892b8fc7905beea7e7c43a"}
Nov 03 19:43:25.148194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:25.141319    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:25.794193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:25.787972    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:26.466519 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:26.466483    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:27.336234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:27.336200    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:28.118792 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:28.115090    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:28.738198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:28.732097    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:28.889496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:28.889463    4531 generic.go:356] "Generic (PLEG): container finished" podID="c75ded06-8e99-4a28-be0b-8de4717a6999" containerID="feb90ff1a2e8944d4085db1d9ca194a87a1af834bfe01ce3cb4008d7125025d1" exitCode=0
Nov 03 19:43:28.889680 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:28.889559    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz" event={"ID":"c75ded06-8e99-4a28-be0b-8de4717a6999","Type":"ContainerDied","Data":"feb90ff1a2e8944d4085db1d9ca194a87a1af834bfe01ce3cb4008d7125025d1"}
Nov 03 19:43:29.412367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.412298    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:29.738532 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.738456    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"]
Nov 03 19:43:29.742716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.742696    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:29.748378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.748351    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"]
Nov 03 19:43:29.795387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.795357    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af73ad23-44vwp\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:29.795551 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.795424    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af73ad23-44vwp\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:29.795551 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.795524    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b84fk\" (UniqueName: \"kubernetes.io/projected/ad958fee-4506-4ec9-b3a2-1f9913595aad-kube-api-access-b84fk\") pod \"maas-api-revoke-sub-af73ad23-44vwp\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:29.896717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.896681    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af73ad23-44vwp\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:29.896930 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.896883    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af73ad23-44vwp\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:29.897081 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.896943    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84fk\" (UniqueName: \"kubernetes.io/projected/ad958fee-4506-4ec9-b3a2-1f9913595aad-kube-api-access-b84fk\") pod \"maas-api-revoke-sub-af73ad23-44vwp\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:29.897532 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.897509    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af73ad23-44vwp\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:29.899197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.899158    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af73ad23-44vwp\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:29.904383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:29.904362    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b84fk\" (UniqueName: \"kubernetes.io/projected/ad958fee-4506-4ec9-b3a2-1f9913595aad-kube-api-access-b84fk\") pod \"maas-api-revoke-sub-af73ad23-44vwp\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:30.047572 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.047550    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:30.073813 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.073766    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:30.099055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.099021    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-xks-ca\") pod \"c75ded06-8e99-4a28-be0b-8de4717a6999\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") "
Nov 03 19:43:30.099206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.099186    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75ded06-8e99-4a28-be0b-8de4717a6999-kube-api-access-qbrfd\" (UniqueName: \"kubernetes.io/projected/c75ded06-8e99-4a28-be0b-8de4717a6999-kube-api-access-qbrfd\") pod \"c75ded06-8e99-4a28-be0b-8de4717a6999\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") "
Nov 03 19:43:30.099289 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.099225    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-openshift-ca\") pod \"c75ded06-8e99-4a28-be0b-8de4717a6999\" (UID: \"c75ded06-8e99-4a28-be0b-8de4717a6999\") "
Nov 03 19:43:30.100056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.100025    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-openshift-ca" pod "c75ded06-8e99-4a28-be0b-8de4717a6999" (UID: "c75ded06-8e99-4a28-be0b-8de4717a6999"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:30.102403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.102372    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75ded06-8e99-4a28-be0b-8de4717a6999-kube-api-access-qbrfd" pod "c75ded06-8e99-4a28-be0b-8de4717a6999" (UID: "c75ded06-8e99-4a28-be0b-8de4717a6999"). InnerVolumeSpecName "kube-api-access-qbrfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:30.102506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.102399    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-xks-ca" pod "c75ded06-8e99-4a28-be0b-8de4717a6999" (UID: "c75ded06-8e99-4a28-be0b-8de4717a6999"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:30.146794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.145869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:30.200640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.200539    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:30.200640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.200581    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbrfd\" (UniqueName: \"kubernetes.io/projected/c75ded06-8e99-4a28-be0b-8de4717a6999-kube-api-access-qbrfd\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:30.200640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.200602    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75ded06-8e99-4a28-be0b-8de4717a6999-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:30.381777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.381751    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"]
Nov 03 19:43:30.382823 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:43:30.382796    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad958fee_4506_4ec9_b3a2_1f9913595aad.slice/crio-34031bdcf1baa90832002ed7daa25ae89b99f6cc2ac67bccda00e70986429c9b WatchSource:0}: Error finding container 34031bdcf1baa90832002ed7daa25ae89b99f6cc2ac67bccda00e70986429c9b: Status 404 returned error can't find the container with id 34031bdcf1baa90832002ed7daa25ae89b99f6cc2ac67bccda00e70986429c9b
Nov 03 19:43:30.784850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.773137    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:30.901574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.901492    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"
Nov 03 19:43:30.901574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.901517    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz" event={"ID":"c75ded06-8e99-4a28-be0b-8de4717a6999","Type":"ContainerDied","Data":"178eeb009b60d3efaea355a45b662eb0de1a104550892b8fc7905beea7e7c43a"}
Nov 03 19:43:30.901574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.901569    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="178eeb009b60d3efaea355a45b662eb0de1a104550892b8fc7905beea7e7c43a"
Nov 03 19:43:30.903302 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.903280    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp" event={"ID":"ad958fee-4506-4ec9-b3a2-1f9913595aad","Type":"ContainerStarted","Data":"2508335227f02d74ac65733875705690595d12075e6b426f3b7c7efbd85b9699"}
Nov 03 19:43:30.903432 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.903318    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp" event={"ID":"ad958fee-4506-4ec9-b3a2-1f9913595aad","Type":"ContainerStarted","Data":"34031bdcf1baa90832002ed7daa25ae89b99f6cc2ac67bccda00e70986429c9b"}
Nov 03 19:43:30.918319 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:30.918281    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp" podStartSLOduration=1.918268286 podStartE2EDuration="1.918268286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:30.916598472 +0000 UTC m=+724.186721767" watchObservedRunningTime="2026-11-03 19:43:30.918268286 +0000 UTC m=+724.188391580"
Nov 03 19:43:31.450250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:31.442834    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:31.909774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:31.909736    4531 generic.go:356] "Generic (PLEG): container finished" podID="ad958fee-4506-4ec9-b3a2-1f9913595aad" containerID="2508335227f02d74ac65733875705690595d12075e6b426f3b7c7efbd85b9699" exitCode=0
Nov 03 19:43:31.910193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:31.909816    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp" event={"ID":"ad958fee-4506-4ec9-b3a2-1f9913595aad","Type":"ContainerDied","Data":"2508335227f02d74ac65733875705690595d12075e6b426f3b7c7efbd85b9699"}
Nov 03 19:43:32.029032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:32.028998    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:32.753200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:32.752143    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:33.057349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.057325    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:33.160636 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.160598    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad958fee-4506-4ec9-b3a2-1f9913595aad-kube-api-access-b84fk\" (UniqueName: \"kubernetes.io/projected/ad958fee-4506-4ec9-b3a2-1f9913595aad-kube-api-access-b84fk\") pod \"ad958fee-4506-4ec9-b3a2-1f9913595aad\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") "
Nov 03 19:43:33.160810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.160678    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-openshift-ca\") pod \"ad958fee-4506-4ec9-b3a2-1f9913595aad\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") "
Nov 03 19:43:33.160810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.160756    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-xks-ca\") pod \"ad958fee-4506-4ec9-b3a2-1f9913595aad\" (UID: \"ad958fee-4506-4ec9-b3a2-1f9913595aad\") "
Nov 03 19:43:33.161044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.161018    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-openshift-ca" pod "ad958fee-4506-4ec9-b3a2-1f9913595aad" (UID: "ad958fee-4506-4ec9-b3a2-1f9913595aad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:33.161163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.161149    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:33.162889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.162862    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad958fee-4506-4ec9-b3a2-1f9913595aad-kube-api-access-b84fk" pod "ad958fee-4506-4ec9-b3a2-1f9913595aad" (UID: "ad958fee-4506-4ec9-b3a2-1f9913595aad"). InnerVolumeSpecName "kube-api-access-b84fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:33.163009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.162931    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-xks-ca" pod "ad958fee-4506-4ec9-b3a2-1f9913595aad" (UID: "ad958fee-4506-4ec9-b3a2-1f9913595aad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:33.262572 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.262535    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b84fk\" (UniqueName: \"kubernetes.io/projected/ad958fee-4506-4ec9-b3a2-1f9913595aad-kube-api-access-b84fk\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:33.262572 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.262565    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad958fee-4506-4ec9-b3a2-1f9913595aad-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:33.408864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.408829    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:33.921568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.921537    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"
Nov 03 19:43:33.921746 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.921573    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp" event={"ID":"ad958fee-4506-4ec9-b3a2-1f9913595aad","Type":"ContainerDied","Data":"34031bdcf1baa90832002ed7daa25ae89b99f6cc2ac67bccda00e70986429c9b"}
Nov 03 19:43:33.921746 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:33.921645    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34031bdcf1baa90832002ed7daa25ae89b99f6cc2ac67bccda00e70986429c9b"
Nov 03 19:43:34.168225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:34.167660    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:34.973891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:34.973855    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:35.769626 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:35.769578    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:36.540628 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:36.540588    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:37.538118 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:37.538063    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:38.554330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:38.551740    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:39.339734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:39.339590    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:39.998197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:39.998140    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"]
Nov 03 19:43:39.998817 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:39.998797    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ad958fee-4506-4ec9-b3a2-1f9913595aad" containerName="revoke-keys"
Nov 03 19:43:39.998910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:39.998818    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad958fee-4506-4ec9-b3a2-1f9913595aad" containerName="revoke-keys"
Nov 03 19:43:39.998910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:39.998834    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c75ded06-8e99-4a28-be0b-8de4717a6999" containerName="revoke-keys"
Nov 03 19:43:39.998910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:39.998841    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75ded06-8e99-4a28-be0b-8de4717a6999" containerName="revoke-keys"
Nov 03 19:43:39.999033 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:39.998984    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad958fee-4506-4ec9-b3a2-1f9913595aad" containerName="revoke-keys"
Nov 03 19:43:39.999033 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:39.998995    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75ded06-8e99-4a28-be0b-8de4717a6999" containerName="revoke-keys"
Nov 03 19:43:40.003683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.003664    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:40.012932 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.012720    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:43:40.019243 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.011353    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:40.023880 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.023822    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"]
Nov 03 19:43:40.060537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.060493    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqxd\" (UniqueName: \"kubernetes.io/projected/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-kube-api-access-bjqxd\") pod \"maas-api-revoke-sub-90922dc8-lwqk8\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:40.060726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.060658    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90922dc8-lwqk8\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:40.060726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.060704    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90922dc8-lwqk8\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:40.110599 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.110566    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:40.167248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.167213    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqxd\" (UniqueName: \"kubernetes.io/projected/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-kube-api-access-bjqxd\") pod \"maas-api-revoke-sub-90922dc8-lwqk8\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:40.167585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.167554    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90922dc8-lwqk8\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:40.167763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.167730    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90922dc8-lwqk8\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:40.168475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.168452    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90922dc8-lwqk8\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:40.171496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.171473    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90922dc8-lwqk8\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:40.174352 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.174329    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqxd\" (UniqueName: \"kubernetes.io/projected/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-kube-api-access-bjqxd\") pod \"maas-api-revoke-sub-90922dc8-lwqk8\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:40.327542 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.327463    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:40.603111 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.603085    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"]
Nov 03 19:43:40.604425 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:43:40.604398    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ba0b5a_0ffe_41a7_8925_a886f068a1b3.slice/crio-7843c20154cdd9c806713aa5c7581740d30642fc219de8366f8450b330bfb2e8 WatchSource:0}: Error finding container 7843c20154cdd9c806713aa5c7581740d30642fc219de8366f8450b330bfb2e8: Status 404 returned error can't find the container with id 7843c20154cdd9c806713aa5c7581740d30642fc219de8366f8450b330bfb2e8
Nov 03 19:43:40.721644 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.721607    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:40.963164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.963095    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8" event={"ID":"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3","Type":"ContainerStarted","Data":"db6fd52270f8a92401cea88a52d19623315e6c7f6f3a9915752e96171901f714"}
Nov 03 19:43:40.963164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:40.963160    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8" event={"ID":"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3","Type":"ContainerStarted","Data":"7843c20154cdd9c806713aa5c7581740d30642fc219de8366f8450b330bfb2e8"}
Nov 03 19:43:41.667204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.665284    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8" podStartSLOduration=2.665238558 podStartE2EDuration="2.665238558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:40.979373963 +0000 UTC m=+734.249497260" watchObservedRunningTime="2026-11-03 19:43:41.665238558 +0000 UTC m=+734.935361856"
Nov 03 19:43:41.675471 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.670505    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"]
Nov 03 19:43:41.723199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.719276    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:41.726770 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.726085    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"]
Nov 03 19:43:41.726770 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.726298    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:41.823234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.818761    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7deb052d-6k674\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:41.823234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.819028    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcg75\" (UniqueName: \"kubernetes.io/projected/40baf39d-7d38-4997-9e5e-9f4f35005183-kube-api-access-xcg75\") pod \"maas-api-revoke-sub-7deb052d-6k674\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:41.823234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.819221    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7deb052d-6k674\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:41.920608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.920505    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcg75\" (UniqueName: \"kubernetes.io/projected/40baf39d-7d38-4997-9e5e-9f4f35005183-kube-api-access-xcg75\") pod \"maas-api-revoke-sub-7deb052d-6k674\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:41.920782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.920713    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7deb052d-6k674\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:41.920782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.920772    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7deb052d-6k674\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:41.921556 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.921527    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7deb052d-6k674\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:41.923231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.923207    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7deb052d-6k674\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:41.927554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.927532    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcg75\" (UniqueName: \"kubernetes.io/projected/40baf39d-7d38-4997-9e5e-9f4f35005183-kube-api-access-xcg75\") pod \"maas-api-revoke-sub-7deb052d-6k674\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:41.973622 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.973579    4531 generic.go:356] "Generic (PLEG): container finished" podID="e4ba0b5a-0ffe-41a7-8925-a886f068a1b3" containerID="db6fd52270f8a92401cea88a52d19623315e6c7f6f3a9915752e96171901f714" exitCode=0
Nov 03 19:43:41.973797 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:41.973678    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8" event={"ID":"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3","Type":"ContainerDied","Data":"db6fd52270f8a92401cea88a52d19623315e6c7f6f3a9915752e96171901f714"}
Nov 03 19:43:42.132395 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:42.132357    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:42.413513 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:42.413484    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"]
Nov 03 19:43:42.416923 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:43:42.416682    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40baf39d_7d38_4997_9e5e_9f4f35005183.slice/crio-313ef19d256fe794c251abe8558d7cd71143750b269b2353144d7c068c1da391 WatchSource:0}: Error finding container 313ef19d256fe794c251abe8558d7cd71143750b269b2353144d7c068c1da391: Status 404 returned error can't find the container with id 313ef19d256fe794c251abe8558d7cd71143750b269b2353144d7c068c1da391
Nov 03 19:43:42.453288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:42.452005    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:42.981416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:42.981372    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674" event={"ID":"40baf39d-7d38-4997-9e5e-9f4f35005183","Type":"ContainerStarted","Data":"1eb7dcebae53c86aea867dc7969d158dbb2affea88b960e70cd4f33e74178d8b"}
Nov 03 19:43:42.981866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:42.981434    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674" event={"ID":"40baf39d-7d38-4997-9e5e-9f4f35005183","Type":"ContainerStarted","Data":"313ef19d256fe794c251abe8558d7cd71143750b269b2353144d7c068c1da391"}
Nov 03 19:43:42.995247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:42.995198    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674" podStartSLOduration=1.995158725 podStartE2EDuration="1.995158725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:42.993521738 +0000 UTC m=+736.263645031" watchObservedRunningTime="2026-11-03 19:43:42.995158725 +0000 UTC m=+736.265282020"
Nov 03 19:43:43.133690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.133618    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:43.185376 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.185353    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:43.240405 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.240376    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-openshift-ca\") pod \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") "
Nov 03 19:43:43.240564 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.240521    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-xks-ca\") pod \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") "
Nov 03 19:43:43.240617 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.240601    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-kube-api-access-bjqxd\" (UniqueName: \"kubernetes.io/projected/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-kube-api-access-bjqxd\") pod \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\" (UID: \"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3\") "
Nov 03 19:43:43.240742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.240719    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-openshift-ca" pod "e4ba0b5a-0ffe-41a7-8925-a886f068a1b3" (UID: "e4ba0b5a-0ffe-41a7-8925-a886f068a1b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:43.241087 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.241061    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:43.242736 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.242709    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-xks-ca" pod "e4ba0b5a-0ffe-41a7-8925-a886f068a1b3" (UID: "e4ba0b5a-0ffe-41a7-8925-a886f068a1b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:43.242842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.242753    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-kube-api-access-bjqxd" pod "e4ba0b5a-0ffe-41a7-8925-a886f068a1b3" (UID: "e4ba0b5a-0ffe-41a7-8925-a886f068a1b3"). InnerVolumeSpecName "kube-api-access-bjqxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:43.342498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.342467    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:43.342498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.342493    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjqxd\" (UniqueName: \"kubernetes.io/projected/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3-kube-api-access-bjqxd\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:43.756973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.756933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:43.989085 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.989049    4531 generic.go:356] "Generic (PLEG): container finished" podID="40baf39d-7d38-4997-9e5e-9f4f35005183" containerID="1eb7dcebae53c86aea867dc7969d158dbb2affea88b960e70cd4f33e74178d8b" exitCode=0
Nov 03 19:43:43.989609 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.989126    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674" event={"ID":"40baf39d-7d38-4997-9e5e-9f4f35005183","Type":"ContainerDied","Data":"1eb7dcebae53c86aea867dc7969d158dbb2affea88b960e70cd4f33e74178d8b"}
Nov 03 19:43:43.990675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.990654    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"
Nov 03 19:43:43.990793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.990679    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8" event={"ID":"e4ba0b5a-0ffe-41a7-8925-a886f068a1b3","Type":"ContainerDied","Data":"7843c20154cdd9c806713aa5c7581740d30642fc219de8366f8450b330bfb2e8"}
Nov 03 19:43:43.990793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:43.990724    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7843c20154cdd9c806713aa5c7581740d30642fc219de8366f8450b330bfb2e8"
Nov 03 19:43:44.532231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:44.525779    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:45.185779 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.185745    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:45.214564 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.214528    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:45.274002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.273920    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-xks-ca\") pod \"40baf39d-7d38-4997-9e5e-9f4f35005183\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") "
Nov 03 19:43:45.274205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.274040    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-openshift-ca\") pod \"40baf39d-7d38-4997-9e5e-9f4f35005183\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") "
Nov 03 19:43:45.274205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.274165    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40baf39d-7d38-4997-9e5e-9f4f35005183-kube-api-access-xcg75\" (UniqueName: \"kubernetes.io/projected/40baf39d-7d38-4997-9e5e-9f4f35005183-kube-api-access-xcg75\") pod \"40baf39d-7d38-4997-9e5e-9f4f35005183\" (UID: \"40baf39d-7d38-4997-9e5e-9f4f35005183\") "
Nov 03 19:43:45.274466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.274434    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-openshift-ca" pod "40baf39d-7d38-4997-9e5e-9f4f35005183" (UID: "40baf39d-7d38-4997-9e5e-9f4f35005183"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:45.274721 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.274703    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:45.276328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.276304    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40baf39d-7d38-4997-9e5e-9f4f35005183-kube-api-access-xcg75" pod "40baf39d-7d38-4997-9e5e-9f4f35005183" (UID: "40baf39d-7d38-4997-9e5e-9f4f35005183"). InnerVolumeSpecName "kube-api-access-xcg75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:45.276422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.276383    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-xks-ca" pod "40baf39d-7d38-4997-9e5e-9f4f35005183" (UID: "40baf39d-7d38-4997-9e5e-9f4f35005183"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:45.375557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.375523    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40baf39d-7d38-4997-9e5e-9f4f35005183-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:45.375557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.375552    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcg75\" (UniqueName: \"kubernetes.io/projected/40baf39d-7d38-4997-9e5e-9f4f35005183-kube-api-access-xcg75\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:45.848527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:45.848482    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:46.005825 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.005794    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"
Nov 03 19:43:46.006033 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.005829    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674" event={"ID":"40baf39d-7d38-4997-9e5e-9f4f35005183","Type":"ContainerDied","Data":"313ef19d256fe794c251abe8558d7cd71143750b269b2353144d7c068c1da391"}
Nov 03 19:43:46.006033 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.005887    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="313ef19d256fe794c251abe8558d7cd71143750b269b2353144d7c068c1da391"
Nov 03 19:43:46.453882 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.443001    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:46.664889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.664856    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"]
Nov 03 19:43:46.665450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.665435    4531 cpu_manager.go:414] "containerMap: removing container" podUID="40baf39d-7d38-4997-9e5e-9f4f35005183" containerName="revoke-keys"
Nov 03 19:43:46.665507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.665451    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40baf39d-7d38-4997-9e5e-9f4f35005183" containerName="revoke-keys"
Nov 03 19:43:46.665507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.665466    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e4ba0b5a-0ffe-41a7-8925-a886f068a1b3" containerName="revoke-keys"
Nov 03 19:43:46.665507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.665472    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ba0b5a-0ffe-41a7-8925-a886f068a1b3" containerName="revoke-keys"
Nov 03 19:43:46.665603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.665560    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ba0b5a-0ffe-41a7-8925-a886f068a1b3" containerName="revoke-keys"
Nov 03 19:43:46.665603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.665576    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="40baf39d-7d38-4997-9e5e-9f4f35005183" containerName="revoke-keys"
Nov 03 19:43:46.669047 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.669026    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:46.672643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.672069    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:46.672643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.672101    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:43:46.675556 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.675531    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"]
Nov 03 19:43:46.691521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.691490    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e189962-pggfx\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:46.691639 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.691614    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qv56\" (UniqueName: \"kubernetes.io/projected/197e6009-fb6b-4432-a8d6-f0cdfeff4987-kube-api-access-2qv56\") pod \"maas-api-revoke-sub-4e189962-pggfx\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:46.691684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.691672    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e189962-pggfx\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:46.793138 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.793046    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e189962-pggfx\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:46.793324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.793141    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qv56\" (UniqueName: \"kubernetes.io/projected/197e6009-fb6b-4432-a8d6-f0cdfeff4987-kube-api-access-2qv56\") pod \"maas-api-revoke-sub-4e189962-pggfx\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:46.793324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.793238    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e189962-pggfx\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:46.793697 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.793670    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e189962-pggfx\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:46.795662 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.795640    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e189962-pggfx\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:46.801126 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.801103    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qv56\" (UniqueName: \"kubernetes.io/projected/197e6009-fb6b-4432-a8d6-f0cdfeff4987-kube-api-access-2qv56\") pod \"maas-api-revoke-sub-4e189962-pggfx\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:46.987421 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:46.987389    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:47.148567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.148411    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:47.263666 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.263635    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"]
Nov 03 19:43:47.265213 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:43:47.265158    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197e6009_fb6b_4432_a8d6_f0cdfeff4987.slice/crio-2bb18cf9ef6827406920c3051d60ed693dcd5d4b5cac5dfab3616422ed77785e WatchSource:0}: Error finding container 2bb18cf9ef6827406920c3051d60ed693dcd5d4b5cac5dfab3616422ed77785e: Status 404 returned error can't find the container with id 2bb18cf9ef6827406920c3051d60ed693dcd5d4b5cac5dfab3616422ed77785e
Nov 03 19:43:47.580389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.580325    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"]
Nov 03 19:43:47.584079 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.584062    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:47.591972 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.591909    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"]
Nov 03 19:43:47.702723 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.702685    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2592cfd8-sj9pt\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:47.702875 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.702797    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2592cfd8-sj9pt\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:47.702942 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.702887    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r86b\" (UniqueName: \"kubernetes.io/projected/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-kube-api-access-8r86b\") pod \"maas-api-revoke-sub-2592cfd8-sj9pt\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:47.803759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.803728    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r86b\" (UniqueName: \"kubernetes.io/projected/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-kube-api-access-8r86b\") pod \"maas-api-revoke-sub-2592cfd8-sj9pt\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:47.803910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.803844    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2592cfd8-sj9pt\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:47.803958 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.803938    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2592cfd8-sj9pt\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:47.804645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.804616    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2592cfd8-sj9pt\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:47.806464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.806440    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2592cfd8-sj9pt\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:47.811276 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.811230    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r86b\" (UniqueName: \"kubernetes.io/projected/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-kube-api-access-8r86b\") pod \"maas-api-revoke-sub-2592cfd8-sj9pt\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:47.902781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.902748    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:47.972452 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:47.972418    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:48.020703 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:48.020670    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx" event={"ID":"197e6009-fb6b-4432-a8d6-f0cdfeff4987","Type":"ContainerStarted","Data":"c82952a670dc73aeb6369b39b48fef67acdece780210900ebf56c90b7481efb2"}
Nov 03 19:43:48.020842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:48.020727    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx" event={"ID":"197e6009-fb6b-4432-a8d6-f0cdfeff4987","Type":"ContainerStarted","Data":"2bb18cf9ef6827406920c3051d60ed693dcd5d4b5cac5dfab3616422ed77785e"}
Nov 03 19:43:48.040099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:48.040035    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx" podStartSLOduration=2.040014654 podStartE2EDuration="2.040014654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:48.034782613 +0000 UTC m=+741.304905908" watchObservedRunningTime="2026-11-03 19:43:48.040014654 +0000 UTC m=+741.310137950"
Nov 03 19:43:48.216881 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:48.216849    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"]
Nov 03 19:43:48.218129 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:43:48.218104    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716a9aa4_ec8a_47cf_bd77_7e616298e2fb.slice/crio-8cb6bd5cc353c9f6e9722b9085f57c4209e5dde19a11c760c178eb4b6964b4a2 WatchSource:0}: Error finding container 8cb6bd5cc353c9f6e9722b9085f57c4209e5dde19a11c760c178eb4b6964b4a2: Status 404 returned error can't find the container with id 8cb6bd5cc353c9f6e9722b9085f57c4209e5dde19a11c760c178eb4b6964b4a2
Nov 03 19:43:48.729703 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:48.728772    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:49.030199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:49.030086    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt" event={"ID":"716a9aa4-ec8a-47cf-bd77-7e616298e2fb","Type":"ContainerStarted","Data":"afc45e5c9d89a7b2135ecdd3de5db58723d08876f06f4b1ecb0bb6884f934072"}
Nov 03 19:43:49.030199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:49.030158    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt" event={"ID":"716a9aa4-ec8a-47cf-bd77-7e616298e2fb","Type":"ContainerStarted","Data":"8cb6bd5cc353c9f6e9722b9085f57c4209e5dde19a11c760c178eb4b6964b4a2"}
Nov 03 19:43:49.032078 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:49.032050    4531 generic.go:356] "Generic (PLEG): container finished" podID="197e6009-fb6b-4432-a8d6-f0cdfeff4987" containerID="c82952a670dc73aeb6369b39b48fef67acdece780210900ebf56c90b7481efb2" exitCode=0
Nov 03 19:43:49.032235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:49.032088    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx" event={"ID":"197e6009-fb6b-4432-a8d6-f0cdfeff4987","Type":"ContainerDied","Data":"c82952a670dc73aeb6369b39b48fef67acdece780210900ebf56c90b7481efb2"}
Nov 03 19:43:49.047396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:49.047327    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt" podStartSLOduration=2.047311722 podStartE2EDuration="2.047311722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:49.045675048 +0000 UTC m=+742.315798343" watchObservedRunningTime="2026-11-03 19:43:49.047311722 +0000 UTC m=+742.317435016"
Nov 03 19:43:49.615706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:49.614099    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:50.041356 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.041321    4531 generic.go:356] "Generic (PLEG): container finished" podID="716a9aa4-ec8a-47cf-bd77-7e616298e2fb" containerID="afc45e5c9d89a7b2135ecdd3de5db58723d08876f06f4b1ecb0bb6884f934072" exitCode=0
Nov 03 19:43:50.041811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.041413    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt" event={"ID":"716a9aa4-ec8a-47cf-bd77-7e616298e2fb","Type":"ContainerDied","Data":"afc45e5c9d89a7b2135ecdd3de5db58723d08876f06f4b1ecb0bb6884f934072"}
Nov 03 19:43:50.209795 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.209770    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:50.324245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.323526    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:50.336926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.336885    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-xks-ca\") pod \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") "
Nov 03 19:43:50.337416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.337389    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-openshift-ca\") pod \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") "
Nov 03 19:43:50.337586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.337572    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197e6009-fb6b-4432-a8d6-f0cdfeff4987-kube-api-access-2qv56\" (UniqueName: \"kubernetes.io/projected/197e6009-fb6b-4432-a8d6-f0cdfeff4987-kube-api-access-2qv56\") pod \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\" (UID: \"197e6009-fb6b-4432-a8d6-f0cdfeff4987\") "
Nov 03 19:43:50.341264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.341233    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197e6009-fb6b-4432-a8d6-f0cdfeff4987-kube-api-access-2qv56" pod "197e6009-fb6b-4432-a8d6-f0cdfeff4987" (UID: "197e6009-fb6b-4432-a8d6-f0cdfeff4987"). InnerVolumeSpecName "kube-api-access-2qv56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:50.342102 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.342042    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-openshift-ca" pod "197e6009-fb6b-4432-a8d6-f0cdfeff4987" (UID: "197e6009-fb6b-4432-a8d6-f0cdfeff4987"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:50.377153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.375337    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-xks-ca" pod "197e6009-fb6b-4432-a8d6-f0cdfeff4987" (UID: "197e6009-fb6b-4432-a8d6-f0cdfeff4987"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:50.439045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.439014    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:50.439045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.439046    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qv56\" (UniqueName: \"kubernetes.io/projected/197e6009-fb6b-4432-a8d6-f0cdfeff4987-kube-api-access-2qv56\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:50.439274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:50.439061    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197e6009-fb6b-4432-a8d6-f0cdfeff4987-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:50.573839 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:43:50.573800    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:43:51.036473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.036437    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:51.076958 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.076926    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"
Nov 03 19:43:51.077458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.076969    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx" event={"ID":"197e6009-fb6b-4432-a8d6-f0cdfeff4987","Type":"ContainerDied","Data":"2bb18cf9ef6827406920c3051d60ed693dcd5d4b5cac5dfab3616422ed77785e"}
Nov 03 19:43:51.077458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.077030    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bb18cf9ef6827406920c3051d60ed693dcd5d4b5cac5dfab3616422ed77785e"
Nov 03 19:43:51.244571 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.244537    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:51.359903 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.359870    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-xks-ca\") pod \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") "
Nov 03 19:43:51.360078 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.359952    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-openshift-ca\") pod \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") "
Nov 03 19:43:51.360119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.360110    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-kube-api-access-8r86b\" (UniqueName: \"kubernetes.io/projected/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-kube-api-access-8r86b\") pod \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\" (UID: \"716a9aa4-ec8a-47cf-bd77-7e616298e2fb\") "
Nov 03 19:43:51.360360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.360333    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-openshift-ca" pod "716a9aa4-ec8a-47cf-bd77-7e616298e2fb" (UID: "716a9aa4-ec8a-47cf-bd77-7e616298e2fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:51.360472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.360457    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:51.362161 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.362132    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-xks-ca" pod "716a9aa4-ec8a-47cf-bd77-7e616298e2fb" (UID: "716a9aa4-ec8a-47cf-bd77-7e616298e2fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:51.362298 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.362222    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-kube-api-access-8r86b" pod "716a9aa4-ec8a-47cf-bd77-7e616298e2fb" (UID: "716a9aa4-ec8a-47cf-bd77-7e616298e2fb"). InnerVolumeSpecName "kube-api-access-8r86b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:51.462035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.461952    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:51.462035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.461991    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r86b\" (UniqueName: \"kubernetes.io/projected/716a9aa4-ec8a-47cf-bd77-7e616298e2fb-kube-api-access-8r86b\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:51.776346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:51.776267    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:52.086319 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.086230    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"
Nov 03 19:43:52.086830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.086221    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt" event={"ID":"716a9aa4-ec8a-47cf-bd77-7e616298e2fb","Type":"ContainerDied","Data":"8cb6bd5cc353c9f6e9722b9085f57c4209e5dde19a11c760c178eb4b6964b4a2"}
Nov 03 19:43:52.086830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.086373    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cb6bd5cc353c9f6e9722b9085f57c4209e5dde19a11c760c178eb4b6964b4a2"
Nov 03 19:43:52.900197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.900056    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"]
Nov 03 19:43:52.904195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.902588    4531 cpu_manager.go:414] "containerMap: removing container" podUID="197e6009-fb6b-4432-a8d6-f0cdfeff4987" containerName="revoke-keys"
Nov 03 19:43:52.904195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.902616    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197e6009-fb6b-4432-a8d6-f0cdfeff4987" containerName="revoke-keys"
Nov 03 19:43:52.904195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.902632    4531 cpu_manager.go:414] "containerMap: removing container" podUID="716a9aa4-ec8a-47cf-bd77-7e616298e2fb" containerName="revoke-keys"
Nov 03 19:43:52.904195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.902640    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716a9aa4-ec8a-47cf-bd77-7e616298e2fb" containerName="revoke-keys"
Nov 03 19:43:52.904195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.902801    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="197e6009-fb6b-4432-a8d6-f0cdfeff4987" containerName="revoke-keys"
Nov 03 19:43:52.904195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.902814    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="716a9aa4-ec8a-47cf-bd77-7e616298e2fb" containerName="revoke-keys"
Nov 03 19:43:52.929197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.919223    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:52.954920 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.949327    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:52.954920 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:52.949462    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:43:53.002643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.001061    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e8caeac-sx5vk\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:53.002643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.001139    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e8caeac-sx5vk\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:53.002643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.001281    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwqzh\" (UniqueName: \"kubernetes.io/projected/64bf8c2c-a57e-47a9-b596-30eda5588e61-kube-api-access-xwqzh\") pod \"maas-api-revoke-sub-1e8caeac-sx5vk\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:53.062192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.050708    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"]
Nov 03 19:43:53.066161 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.066116    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:53.103379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.103338    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e8caeac-sx5vk\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:53.104000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.103978    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e8caeac-sx5vk\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:53.104221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.104199    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqzh\" (UniqueName: \"kubernetes.io/projected/64bf8c2c-a57e-47a9-b596-30eda5588e61-kube-api-access-xwqzh\") pod \"maas-api-revoke-sub-1e8caeac-sx5vk\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:53.113046 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.113018    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e8caeac-sx5vk\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:53.121606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.121578    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e8caeac-sx5vk\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:53.125580 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.125553    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwqzh\" (UniqueName: \"kubernetes.io/projected/64bf8c2c-a57e-47a9-b596-30eda5588e61-kube-api-access-xwqzh\") pod \"maas-api-revoke-sub-1e8caeac-sx5vk\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:53.365019 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.364921    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:53.677648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:53.677301    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"]
Nov 03 19:43:53.679127 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:43:53.679087    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bf8c2c_a57e_47a9_b596_30eda5588e61.slice/crio-68d7de7098d70f95f558d9b2a50eb1c91af03dce46e08208f8a94672b14e7195 WatchSource:0}: Error finding container 68d7de7098d70f95f558d9b2a50eb1c91af03dce46e08208f8a94672b14e7195: Status 404 returned error can't find the container with id 68d7de7098d70f95f558d9b2a50eb1c91af03dce46e08208f8a94672b14e7195
Nov 03 19:43:54.608200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:54.596465    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk" event={"ID":"64bf8c2c-a57e-47a9-b596-30eda5588e61","Type":"ContainerStarted","Data":"68d7de7098d70f95f558d9b2a50eb1c91af03dce46e08208f8a94672b14e7195"}
Nov 03 19:43:54.619481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:54.619443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:55.112199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:55.110363    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:55.416904 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:55.416344    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:55.532814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:55.532276    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:43:55.543067 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:55.542965    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:55.618427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:55.617805    4531 generic.go:356] "Generic (PLEG): container finished" podID="64bf8c2c-a57e-47a9-b596-30eda5588e61" containerID="0bc9910f35d4fdb665c69566014af6e49c194c6dada9bb8f64cd4fae48125b67" exitCode=0
Nov 03 19:43:55.618427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:55.617973    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk" event={"ID":"64bf8c2c-a57e-47a9-b596-30eda5588e61","Type":"ContainerDied","Data":"0bc9910f35d4fdb665c69566014af6e49c194c6dada9bb8f64cd4fae48125b67"}
Nov 03 19:43:56.254808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.254765    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:56.413208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.412915    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:56.774400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.774372    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:56.891027 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.890990    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bf8c2c-a57e-47a9-b596-30eda5588e61-kube-api-access-xwqzh\" (UniqueName: \"kubernetes.io/projected/64bf8c2c-a57e-47a9-b596-30eda5588e61-kube-api-access-xwqzh\") pod \"64bf8c2c-a57e-47a9-b596-30eda5588e61\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") "
Nov 03 19:43:56.891257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.891054    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-xks-ca\") pod \"64bf8c2c-a57e-47a9-b596-30eda5588e61\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") "
Nov 03 19:43:56.891257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.891203    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-openshift-ca\") pod \"64bf8c2c-a57e-47a9-b596-30eda5588e61\" (UID: \"64bf8c2c-a57e-47a9-b596-30eda5588e61\") "
Nov 03 19:43:56.891485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.891466    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-openshift-ca" pod "64bf8c2c-a57e-47a9-b596-30eda5588e61" (UID: "64bf8c2c-a57e-47a9-b596-30eda5588e61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:56.891826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.891804    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:56.893269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.893244    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-xks-ca" pod "64bf8c2c-a57e-47a9-b596-30eda5588e61" (UID: "64bf8c2c-a57e-47a9-b596-30eda5588e61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:56.893372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.893347    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bf8c2c-a57e-47a9-b596-30eda5588e61-kube-api-access-xwqzh" pod "64bf8c2c-a57e-47a9-b596-30eda5588e61" (UID: "64bf8c2c-a57e-47a9-b596-30eda5588e61"). InnerVolumeSpecName "kube-api-access-xwqzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:56.993026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.992996    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwqzh\" (UniqueName: \"kubernetes.io/projected/64bf8c2c-a57e-47a9-b596-30eda5588e61-kube-api-access-xwqzh\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:56.993026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:56.993026    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf8c2c-a57e-47a9-b596-30eda5588e61-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:57.120244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.120160    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:57.639016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.638987    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"
Nov 03 19:43:57.639211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.638980    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk" event={"ID":"64bf8c2c-a57e-47a9-b596-30eda5588e61","Type":"ContainerDied","Data":"68d7de7098d70f95f558d9b2a50eb1c91af03dce46e08208f8a94672b14e7195"}
Nov 03 19:43:57.639211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.639141    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68d7de7098d70f95f558d9b2a50eb1c91af03dce46e08208f8a94672b14e7195"
Nov 03 19:43:57.826264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.826226    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:57.854802 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.854768    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"]
Nov 03 19:43:57.855790 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.855754    4531 cpu_manager.go:414] "containerMap: removing container" podUID="64bf8c2c-a57e-47a9-b596-30eda5588e61" containerName="revoke-keys"
Nov 03 19:43:57.855790 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.855786    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bf8c2c-a57e-47a9-b596-30eda5588e61" containerName="revoke-keys"
Nov 03 19:43:57.856003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.855971    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bf8c2c-a57e-47a9-b596-30eda5588e61" containerName="revoke-keys"
Nov 03 19:43:57.860784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.860759    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:43:57.861551 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.861481    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"]
Nov 03 19:43:57.863453 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.863427    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:57.863659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.863636    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:43:57.923355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.923276    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9cd27fa-nhkzs\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:43:57.923355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.923315    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9cd27fa-nhkzs\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:43:57.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:57.923491    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvdc6\" (UniqueName: \"kubernetes.io/projected/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-kube-api-access-nvdc6\") pod \"maas-api-revoke-sub-a9cd27fa-nhkzs\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:43:58.006533 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.006499    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"]
Nov 03 19:43:58.011102 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.011075    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:43:58.016614 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.016590    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"]
Nov 03 19:43:58.025734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.025702    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvdc6\" (UniqueName: \"kubernetes.io/projected/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-kube-api-access-nvdc6\") pod \"maas-api-revoke-sub-a9cd27fa-nhkzs\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:43:58.025908 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.025887    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9cd27fa-nhkzs\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:43:58.026014 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.025932    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9cd27fa-nhkzs\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:43:58.026831 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.026807    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9cd27fa-nhkzs\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:43:58.028921 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.028889    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9cd27fa-nhkzs\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:43:58.033103 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.033075    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvdc6\" (UniqueName: \"kubernetes.io/projected/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-kube-api-access-nvdc6\") pod \"maas-api-revoke-sub-a9cd27fa-nhkzs\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:43:58.127890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.127855    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaefc7b4-v54zq\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:43:58.128092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.127912    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8jbl\" (UniqueName: \"kubernetes.io/projected/1eae4dea-9ca7-427b-90e2-683861bba5c1-kube-api-access-c8jbl\") pod \"maas-api-revoke-sub-aaefc7b4-v54zq\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:43:58.128092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.128039    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaefc7b4-v54zq\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:43:58.178976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.178901    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:43:58.231032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.229704    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaefc7b4-v54zq\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:43:58.231032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.230318    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8jbl\" (UniqueName: \"kubernetes.io/projected/1eae4dea-9ca7-427b-90e2-683861bba5c1-kube-api-access-c8jbl\") pod \"maas-api-revoke-sub-aaefc7b4-v54zq\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:43:58.231032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.230388    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaefc7b4-v54zq\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:43:58.231032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.230600    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaefc7b4-v54zq\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:43:58.233308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.233282    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaefc7b4-v54zq\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:43:58.237017 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.236987    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8jbl\" (UniqueName: \"kubernetes.io/projected/1eae4dea-9ca7-427b-90e2-683861bba5c1-kube-api-access-c8jbl\") pod \"maas-api-revoke-sub-aaefc7b4-v54zq\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:43:58.329014 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.328983    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:43:58.454297 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.454266    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"]
Nov 03 19:43:58.455676 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:43:58.455646    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62e9b3d_e0ed_4fbc_9a2c_a22fdc9ba84b.slice/crio-2e33f418ebed44b8d9834a1877e6e41104f62d9efb10218137cb3d96f7c417df WatchSource:0}: Error finding container 2e33f418ebed44b8d9834a1877e6e41104f62d9efb10218137cb3d96f7c417df: Status 404 returned error can't find the container with id 2e33f418ebed44b8d9834a1877e6e41104f62d9efb10218137cb3d96f7c417df
Nov 03 19:43:58.483586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.483396    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:58.605168 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.605135    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"]
Nov 03 19:43:58.606350 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:43:58.606322    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eae4dea_9ca7_427b_90e2_683861bba5c1.slice/crio-f83ebc2ce0458fc70e2ff45207c32032ee562583f68693bc439a3837ff26f61b WatchSource:0}: Error finding container f83ebc2ce0458fc70e2ff45207c32032ee562583f68693bc439a3837ff26f61b: Status 404 returned error can't find the container with id f83ebc2ce0458fc70e2ff45207c32032ee562583f68693bc439a3837ff26f61b
Nov 03 19:43:58.646371 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.646344    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq" event={"ID":"1eae4dea-9ca7-427b-90e2-683861bba5c1","Type":"ContainerStarted","Data":"f83ebc2ce0458fc70e2ff45207c32032ee562583f68693bc439a3837ff26f61b"}
Nov 03 19:43:58.647619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:58.647595    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs" event={"ID":"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b","Type":"ContainerStarted","Data":"2e33f418ebed44b8d9834a1877e6e41104f62d9efb10218137cb3d96f7c417df"}
Nov 03 19:43:59.164359 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:59.164324    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:59.655978 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:59.655946    4531 generic.go:356] "Generic (PLEG): container finished" podID="b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b" containerID="1bce39b48744e3ed182df69b30f8253a5b6e0fa09260de9110d60b7128200384" exitCode=0
Nov 03 19:43:59.656112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:59.656039    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs" event={"ID":"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b","Type":"ContainerDied","Data":"1bce39b48744e3ed182df69b30f8253a5b6e0fa09260de9110d60b7128200384"}
Nov 03 19:43:59.657748 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:59.657725    4531 generic.go:356] "Generic (PLEG): container finished" podID="1eae4dea-9ca7-427b-90e2-683861bba5c1" containerID="72203e10b28b33de423f8682a232e37278788b4823932c404873bf4b4758c633" exitCode=0
Nov 03 19:43:59.657872 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:59.657770    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq" event={"ID":"1eae4dea-9ca7-427b-90e2-683861bba5c1","Type":"ContainerDied","Data":"72203e10b28b33de423f8682a232e37278788b4823932c404873bf4b4758c633"}
Nov 03 19:43:59.856696 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:43:59.856610    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:00.649558 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:00.649519    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:00.866928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:00.866900    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:44:00.900032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:00.899971    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:44:00.989731 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:00.989698    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eae4dea-9ca7-427b-90e2-683861bba5c1-kube-api-access-c8jbl\" (UniqueName: \"kubernetes.io/projected/1eae4dea-9ca7-427b-90e2-683861bba5c1-kube-api-access-c8jbl\") pod \"1eae4dea-9ca7-427b-90e2-683861bba5c1\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") "
Nov 03 19:44:00.989928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:00.989762    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-xks-ca\") pod \"1eae4dea-9ca7-427b-90e2-683861bba5c1\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") "
Nov 03 19:44:00.989928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:00.989869    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-openshift-ca\") pod \"1eae4dea-9ca7-427b-90e2-683861bba5c1\" (UID: \"1eae4dea-9ca7-427b-90e2-683861bba5c1\") "
Nov 03 19:44:00.990246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:00.990222    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-openshift-ca" pod "1eae4dea-9ca7-427b-90e2-683861bba5c1" (UID: "1eae4dea-9ca7-427b-90e2-683861bba5c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:00.990493 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:00.990467    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:00.991990 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:00.991966    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-xks-ca" pod "1eae4dea-9ca7-427b-90e2-683861bba5c1" (UID: "1eae4dea-9ca7-427b-90e2-683861bba5c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:00.992083 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:00.991984    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eae4dea-9ca7-427b-90e2-683861bba5c1-kube-api-access-c8jbl" pod "1eae4dea-9ca7-427b-90e2-683861bba5c1" (UID: "1eae4dea-9ca7-427b-90e2-683861bba5c1"). InnerVolumeSpecName "kube-api-access-c8jbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:01.091541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.091507    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-xks-ca\") pod \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") "
Nov 03 19:44:01.091716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.091705    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-openshift-ca\") pod \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") "
Nov 03 19:44:01.091771 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.091729    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-kube-api-access-nvdc6\" (UniqueName: \"kubernetes.io/projected/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-kube-api-access-nvdc6\") pod \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\" (UID: \"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b\") "
Nov 03 19:44:01.092089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.092064    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-openshift-ca" pod "b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b" (UID: "b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:01.092089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.092090    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eae4dea-9ca7-427b-90e2-683861bba5c1-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:01.092320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.092108    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8jbl\" (UniqueName: \"kubernetes.io/projected/1eae4dea-9ca7-427b-90e2-683861bba5c1-kube-api-access-c8jbl\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:01.093700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.093674    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-xks-ca" pod "b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b" (UID: "b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:01.093970 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.093951    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-kube-api-access-nvdc6" pod "b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b" (UID: "b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b"). InnerVolumeSpecName "kube-api-access-nvdc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:01.193810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.193735    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:01.193810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.193762    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvdc6\" (UniqueName: \"kubernetes.io/projected/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-kube-api-access-nvdc6\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:01.193810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.193774    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:01.417370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.417338    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:01.709511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.709474    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"
Nov 03 19:44:01.709511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.709500    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq" event={"ID":"1eae4dea-9ca7-427b-90e2-683861bba5c1","Type":"ContainerDied","Data":"f83ebc2ce0458fc70e2ff45207c32032ee562583f68693bc439a3837ff26f61b"}
Nov 03 19:44:01.709971 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.709550    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f83ebc2ce0458fc70e2ff45207c32032ee562583f68693bc439a3837ff26f61b"
Nov 03 19:44:01.711010 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.710979    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"
Nov 03 19:44:01.711143 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.711038    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs" event={"ID":"b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b","Type":"ContainerDied","Data":"2e33f418ebed44b8d9834a1877e6e41104f62d9efb10218137cb3d96f7c417df"}
Nov 03 19:44:01.711143 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:01.711078    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e33f418ebed44b8d9834a1877e6e41104f62d9efb10218137cb3d96f7c417df"
Nov 03 19:44:03.084199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:03.083262    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:03.289198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:03.288293    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:03.348257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:03.347259    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:03.509329 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:03.509270    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:03.561379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:03.561316    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:04.161752 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.161687    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:04.289206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.289158    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"]
Nov 03 19:44:04.298197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.290218    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b" containerName="revoke-keys"
Nov 03 19:44:04.300693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.300664    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b" containerName="revoke-keys"
Nov 03 19:44:04.301046 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.301030    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1eae4dea-9ca7-427b-90e2-683861bba5c1" containerName="revoke-keys"
Nov 03 19:44:04.301190 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.301164    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eae4dea-9ca7-427b-90e2-683861bba5c1" containerName="revoke-keys"
Nov 03 19:44:04.301804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.301784    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b" containerName="revoke-keys"
Nov 03 19:44:04.302037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.302022    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eae4dea-9ca7-427b-90e2-683861bba5c1" containerName="revoke-keys"
Nov 03 19:44:04.313223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.300478    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:04.316426 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.316389    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:04.329215 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.323387    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:04.349204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.334280    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:04.349204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.338066    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:44:04.349204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.338142    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:44:04.375198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.350298    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:04.375880 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.375847    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-326dc398-hfjt8\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:04.376320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.376297    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfrz6\" (UniqueName: \"kubernetes.io/projected/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-kube-api-access-cfrz6\") pod \"maas-api-revoke-sub-326dc398-hfjt8\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:04.376537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.376517    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-326dc398-hfjt8\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:04.520259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.519870    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:44:04.520259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.519923    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"]
Nov 03 19:44:04.524201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.521025    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:04.524201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.521860    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:44:04.524201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.523037    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfrz6\" (UniqueName: \"kubernetes.io/projected/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-kube-api-access-cfrz6\") pod \"maas-api-revoke-sub-326dc398-hfjt8\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:04.524201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.523074    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-326dc398-hfjt8\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:04.524201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.523137    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-326dc398-hfjt8\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:04.524201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.524111    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-326dc398-hfjt8\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:04.602198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.567268    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:44:04.602198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.567843    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:44:04.602198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.580260    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:04.648208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.644406    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"]
Nov 03 19:44:04.707202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:04.700952    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:05.012425 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.012382    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-326dc398-hfjt8\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:05.029200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.028014    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"]
Nov 03 19:44:05.085781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.085732    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 03 19:44:05.125026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.117019    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7868645a-mhd42\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:05.125026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.117319    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7868645a-mhd42\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:05.125026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.117422    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9n9v\" (UniqueName: \"kubernetes.io/projected/cff74c7d-b1eb-4ed2-b150-b058cc95d542-kube-api-access-h9n9v\") pod \"maas-api-revoke-sub-7868645a-mhd42\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:05.135333 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.132792    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:44:05.179208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.163649    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfrz6\" (UniqueName: \"kubernetes.io/projected/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-kube-api-access-cfrz6\") pod \"maas-api-revoke-sub-326dc398-hfjt8\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:05.260227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.259375    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7868645a-mhd42\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:05.260227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.259468    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7868645a-mhd42\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:05.260227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.259615    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9n9v\" (UniqueName: \"kubernetes.io/projected/cff74c7d-b1eb-4ed2-b150-b058cc95d542-kube-api-access-h9n9v\") pod \"maas-api-revoke-sub-7868645a-mhd42\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:05.321748 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.321667    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7868645a-mhd42\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:05.326348 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.326140    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:05.328504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.328476    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:44:05.349869 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.328813    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:05.349869 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.329771    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:44:05.481202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.478270    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:05.551390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.551079    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7868645a-mhd42\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:05.563589 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.555896    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:44:05.563589 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.556012    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:44:05.563589 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.556049    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:44:05.563589 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.556087    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:44:05.576321 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.565247    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:05.588209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.581056    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:05.588209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.581855    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:44:05.588209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.583213    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:05.589028 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.589004    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9n9v\" (UniqueName: \"kubernetes.io/projected/cff74c7d-b1eb-4ed2-b150-b058cc95d542-kube-api-access-h9n9v\") pod \"maas-api-revoke-sub-7868645a-mhd42\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:05.611486 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.611460    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:44:05.612267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.612246    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:44:05.629629 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.629390    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:05.716373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.715309    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:44:05.765717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.765683    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:05.841637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.841051    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:44:05.872516 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.872493    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:44:05.914388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:05.914338    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:44:06.121844 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:06.121554    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"]
Nov 03 19:44:06.128949 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:06.128922    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff74c7d_b1eb_4ed2_b150_b058cc95d542.slice/crio-ebab957f0070ca6c30822c95effe279e3f640538980dcd1032b54525a357497d WatchSource:0}: Error finding container ebab957f0070ca6c30822c95effe279e3f640538980dcd1032b54525a357497d: Status 404 returned error can't find the container with id ebab957f0070ca6c30822c95effe279e3f640538980dcd1032b54525a357497d
Nov 03 19:44:06.136398 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:06.136374    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"]
Nov 03 19:44:06.136716 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:06.136692    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae49a5f_2faf_4388_b697_8d6dcccb4f97.slice/crio-f1203aaf2c0608972f09dd97c9c5be2a625e1e440f7b6eaee217781b3657439b WatchSource:0}: Error finding container f1203aaf2c0608972f09dd97c9c5be2a625e1e440f7b6eaee217781b3657439b: Status 404 returned error can't find the container with id f1203aaf2c0608972f09dd97c9c5be2a625e1e440f7b6eaee217781b3657439b
Nov 03 19:44:06.510334 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:06.510293    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8" event={"ID":"2ae49a5f-2faf-4388-b697-8d6dcccb4f97","Type":"ContainerStarted","Data":"de44e64ee812fb7f859de31f4d6e38c244d82adb2c07b8cc9cdd70fc903b9209"}
Nov 03 19:44:06.510759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:06.510349    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8" event={"ID":"2ae49a5f-2faf-4388-b697-8d6dcccb4f97","Type":"ContainerStarted","Data":"f1203aaf2c0608972f09dd97c9c5be2a625e1e440f7b6eaee217781b3657439b"}
Nov 03 19:44:06.512042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:06.512002    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42" event={"ID":"cff74c7d-b1eb-4ed2-b150-b058cc95d542","Type":"ContainerStarted","Data":"0aead45c81b634d99e1117fdca48ab157e607779c485d2cfd9adf79eb640dfa2"}
Nov 03 19:44:06.512209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:06.512060    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42" event={"ID":"cff74c7d-b1eb-4ed2-b150-b058cc95d542","Type":"ContainerStarted","Data":"ebab957f0070ca6c30822c95effe279e3f640538980dcd1032b54525a357497d"}
Nov 03 19:44:06.528039 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:06.527996    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8" podStartSLOduration=2.527982279 podStartE2EDuration="2.527982279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:06.525348759 +0000 UTC m=+759.795472054" watchObservedRunningTime="2026-11-03 19:44:06.527982279 +0000 UTC m=+759.798105574"
Nov 03 19:44:06.539946 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:06.539298    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42" podStartSLOduration=2.539284687 podStartE2EDuration="2.539284687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:06.53887377 +0000 UTC m=+759.808997065" watchObservedRunningTime="2026-11-03 19:44:06.539284687 +0000 UTC m=+759.809407983"
Nov 03 19:44:06.593202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:06.582353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:07.442196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:07.441492    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:07.520100 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:07.520064    4531 generic.go:356] "Generic (PLEG): container finished" podID="cff74c7d-b1eb-4ed2-b150-b058cc95d542" containerID="0aead45c81b634d99e1117fdca48ab157e607779c485d2cfd9adf79eb640dfa2" exitCode=0
Nov 03 19:44:07.520552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:07.520151    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42" event={"ID":"cff74c7d-b1eb-4ed2-b150-b058cc95d542","Type":"ContainerDied","Data":"0aead45c81b634d99e1117fdca48ab157e607779c485d2cfd9adf79eb640dfa2"}
Nov 03 19:44:07.521906 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:07.521878    4531 generic.go:356] "Generic (PLEG): container finished" podID="2ae49a5f-2faf-4388-b697-8d6dcccb4f97" containerID="de44e64ee812fb7f859de31f4d6e38c244d82adb2c07b8cc9cdd70fc903b9209" exitCode=0
Nov 03 19:44:07.522038 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:07.521963    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8" event={"ID":"2ae49a5f-2faf-4388-b697-8d6dcccb4f97","Type":"ContainerDied","Data":"de44e64ee812fb7f859de31f4d6e38c244d82adb2c07b8cc9cdd70fc903b9209"}
Nov 03 19:44:08.359798 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.359763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:08.676869 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.676844    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:08.711479 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.711445    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:08.816547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.816506    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-openshift-ca\") pod \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") "
Nov 03 19:44:08.816547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.816552    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff74c7d-b1eb-4ed2-b150-b058cc95d542-kube-api-access-h9n9v\" (UniqueName: \"kubernetes.io/projected/cff74c7d-b1eb-4ed2-b150-b058cc95d542-kube-api-access-h9n9v\") pod \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") "
Nov 03 19:44:08.816786 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.816697    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-openshift-ca\") pod \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") "
Nov 03 19:44:08.816786 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.816725    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-xks-ca\") pod \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\" (UID: \"cff74c7d-b1eb-4ed2-b150-b058cc95d542\") "
Nov 03 19:44:08.816902 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.816819    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-kube-api-access-cfrz6\" (UniqueName: \"kubernetes.io/projected/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-kube-api-access-cfrz6\") pod \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") "
Nov 03 19:44:08.816902 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.816864    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-xks-ca\") pod \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\" (UID: \"2ae49a5f-2faf-4388-b697-8d6dcccb4f97\") "
Nov 03 19:44:08.816902 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.816872    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-openshift-ca" pod "cff74c7d-b1eb-4ed2-b150-b058cc95d542" (UID: "cff74c7d-b1eb-4ed2-b150-b058cc95d542"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:08.817103 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.817082    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-openshift-ca" pod "2ae49a5f-2faf-4388-b697-8d6dcccb4f97" (UID: "2ae49a5f-2faf-4388-b697-8d6dcccb4f97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:08.817376 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.817356    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:08.817376 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.817375    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:08.818861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.818835    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-xks-ca" pod "cff74c7d-b1eb-4ed2-b150-b058cc95d542" (UID: "cff74c7d-b1eb-4ed2-b150-b058cc95d542"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:08.818980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.818927    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-xks-ca" pod "2ae49a5f-2faf-4388-b697-8d6dcccb4f97" (UID: "2ae49a5f-2faf-4388-b697-8d6dcccb4f97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:08.819229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.819204    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff74c7d-b1eb-4ed2-b150-b058cc95d542-kube-api-access-h9n9v" pod "cff74c7d-b1eb-4ed2-b150-b058cc95d542" (UID: "cff74c7d-b1eb-4ed2-b150-b058cc95d542"). InnerVolumeSpecName "kube-api-access-h9n9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:08.819309 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.819210    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-kube-api-access-cfrz6" pod "2ae49a5f-2faf-4388-b697-8d6dcccb4f97" (UID: "2ae49a5f-2faf-4388-b697-8d6dcccb4f97"). InnerVolumeSpecName "kube-api-access-cfrz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:08.918785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.918704    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfrz6\" (UniqueName: \"kubernetes.io/projected/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-kube-api-access-cfrz6\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:08.918785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.918738    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae49a5f-2faf-4388-b697-8d6dcccb4f97-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:08.918785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.918752    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9n9v\" (UniqueName: \"kubernetes.io/projected/cff74c7d-b1eb-4ed2-b150-b058cc95d542-kube-api-access-h9n9v\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:08.918785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.918765    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff74c7d-b1eb-4ed2-b150-b058cc95d542-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:08.947481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:08.947454    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:09.240340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:09.240255    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:09.536182 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:09.536058    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"
Nov 03 19:44:09.536182 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:09.536093    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8" event={"ID":"2ae49a5f-2faf-4388-b697-8d6dcccb4f97","Type":"ContainerDied","Data":"f1203aaf2c0608972f09dd97c9c5be2a625e1e440f7b6eaee217781b3657439b"}
Nov 03 19:44:09.536182 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:09.536152    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1203aaf2c0608972f09dd97c9c5be2a625e1e440f7b6eaee217781b3657439b"
Nov 03 19:44:09.537568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:09.537546    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"
Nov 03 19:44:09.537568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:09.537556    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42" event={"ID":"cff74c7d-b1eb-4ed2-b150-b058cc95d542","Type":"ContainerDied","Data":"ebab957f0070ca6c30822c95effe279e3f640538980dcd1032b54525a357497d"}
Nov 03 19:44:09.537732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:09.537602    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebab957f0070ca6c30822c95effe279e3f640538980dcd1032b54525a357497d"
Nov 03 19:44:09.862670 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:09.840863    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:10.568418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:10.568380    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:11.031450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.031416    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"]
Nov 03 19:44:11.032018 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.032001    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2ae49a5f-2faf-4388-b697-8d6dcccb4f97" containerName="revoke-keys"
Nov 03 19:44:11.032084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.032019    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae49a5f-2faf-4388-b697-8d6dcccb4f97" containerName="revoke-keys"
Nov 03 19:44:11.032084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.032034    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cff74c7d-b1eb-4ed2-b150-b058cc95d542" containerName="revoke-keys"
Nov 03 19:44:11.032084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.032038    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff74c7d-b1eb-4ed2-b150-b058cc95d542" containerName="revoke-keys"
Nov 03 19:44:11.032239 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.032126    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae49a5f-2faf-4388-b697-8d6dcccb4f97" containerName="revoke-keys"
Nov 03 19:44:11.032239 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.032136    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff74c7d-b1eb-4ed2-b150-b058cc95d542" containerName="revoke-keys"
Nov 03 19:44:11.036077 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.036058    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:11.041674 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.041643    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:44:11.041809 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.041691    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:44:11.041861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.041804    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"]
Nov 03 19:44:11.069166 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.069137    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whcpz\" (UniqueName: \"kubernetes.io/projected/e7947c08-1f0f-49d8-a23b-53dd6d666355-kube-api-access-whcpz\") pod \"maas-api-revoke-sub-4f50d8f7-hpb9t\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:11.069349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.069203    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f50d8f7-hpb9t\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:11.069349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.069282    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f50d8f7-hpb9t\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:11.170499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.170465    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whcpz\" (UniqueName: \"kubernetes.io/projected/e7947c08-1f0f-49d8-a23b-53dd6d666355-kube-api-access-whcpz\") pod \"maas-api-revoke-sub-4f50d8f7-hpb9t\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:11.170663 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.170517    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f50d8f7-hpb9t\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:11.170663 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.170552    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f50d8f7-hpb9t\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:11.171210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.171148    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f50d8f7-hpb9t\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:11.172987 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.172963    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f50d8f7-hpb9t\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:11.177035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.177009    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whcpz\" (UniqueName: \"kubernetes.io/projected/e7947c08-1f0f-49d8-a23b-53dd6d666355-kube-api-access-whcpz\") pod \"maas-api-revoke-sub-4f50d8f7-hpb9t\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:11.353414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.353326    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:11.411197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.409326    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:11.665166 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:11.665134    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"]
Nov 03 19:44:11.666316 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:11.666286    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7947c08_1f0f_49d8_a23b_53dd6d666355.slice/crio-5672e14bbf54be78a6fc806b087318417c1f0333999617341c28c7b08c6a6572 WatchSource:0}: Error finding container 5672e14bbf54be78a6fc806b087318417c1f0333999617341c28c7b08c6a6572: Status 404 returned error can't find the container with id 5672e14bbf54be78a6fc806b087318417c1f0333999617341c28c7b08c6a6572
Nov 03 19:44:12.115472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:12.115419    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:12.591827 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:12.591797    4531 generic.go:356] "Generic (PLEG): container finished" podID="e7947c08-1f0f-49d8-a23b-53dd6d666355" containerID="6a073ece546657652cc390019c8cfbd9225b74ba97a25b76ecc2b1641dabd35a" exitCode=0
Nov 03 19:44:12.592003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:12.591837    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t" event={"ID":"e7947c08-1f0f-49d8-a23b-53dd6d666355","Type":"ContainerDied","Data":"6a073ece546657652cc390019c8cfbd9225b74ba97a25b76ecc2b1641dabd35a"}
Nov 03 19:44:12.592003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:12.591908    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t" event={"ID":"e7947c08-1f0f-49d8-a23b-53dd6d666355","Type":"ContainerStarted","Data":"5672e14bbf54be78a6fc806b087318417c1f0333999617341c28c7b08c6a6572"}
Nov 03 19:44:12.739605 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:12.738100    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:13.215601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.215568    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"]
Nov 03 19:44:13.219732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.219706    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.222737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.222713    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0486e8\"/\"distinct-w5-0486e8-kserve-self-signed-certs\""
Nov 03 19:44:13.227699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.227675    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"]
Nov 03 19:44:13.295460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.295430    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-home\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.295610 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.295468    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-dshm\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.295610 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.295587    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-tmp-dir\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.295740 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.295647    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24bpn\" (UniqueName: \"kubernetes.io/projected/e72488d0-dae8-4b0f-9c8c-9695319069b5-kube-api-access-24bpn\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.295799 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.295764    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-model-cache\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.295856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.295807    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72488d0-dae8-4b0f-9c8c-9695319069b5-tls-certs\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.304256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.304230    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:13.403197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.402993    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-home\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.403197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.403052    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-dshm\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.403197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.403153    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-tmp-dir\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.403509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.403251    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24bpn\" (UniqueName: \"kubernetes.io/projected/e72488d0-dae8-4b0f-9c8c-9695319069b5-kube-api-access-24bpn\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.403509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.403308    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-model-cache\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.403509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.403338    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72488d0-dae8-4b0f-9c8c-9695319069b5-tls-certs\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.412273 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.411619    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-tmp-dir\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.412273 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.411880    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-home\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.427195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.422850    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-model-cache\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.437262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.432507    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-dshm\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.452242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.452202    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:13.458193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.455754    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72488d0-dae8-4b0f-9c8c-9695319069b5-tls-certs\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.462793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.458770    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24bpn\" (UniqueName: \"kubernetes.io/projected/e72488d0-dae8-4b0f-9c8c-9695319069b5-kube-api-access-24bpn\") pod \"distinct-w5-0486e8-kserve-76d9c7596c-k9w5f\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.533694 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.533602    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:13.757357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.757334    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:13.808902 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.808817    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-openshift-ca\") pod \"e7947c08-1f0f-49d8-a23b-53dd6d666355\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") "
Nov 03 19:44:13.809060 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.808980    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-xks-ca\") pod \"e7947c08-1f0f-49d8-a23b-53dd6d666355\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") "
Nov 03 19:44:13.809060 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.809012    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7947c08-1f0f-49d8-a23b-53dd6d666355-kube-api-access-whcpz\" (UniqueName: \"kubernetes.io/projected/e7947c08-1f0f-49d8-a23b-53dd6d666355-kube-api-access-whcpz\") pod \"e7947c08-1f0f-49d8-a23b-53dd6d666355\" (UID: \"e7947c08-1f0f-49d8-a23b-53dd6d666355\") "
Nov 03 19:44:13.809324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.809277    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-openshift-ca" pod "e7947c08-1f0f-49d8-a23b-53dd6d666355" (UID: "e7947c08-1f0f-49d8-a23b-53dd6d666355"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:13.811393 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.811371    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-xks-ca" pod "e7947c08-1f0f-49d8-a23b-53dd6d666355" (UID: "e7947c08-1f0f-49d8-a23b-53dd6d666355"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:13.811575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.811556    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7947c08-1f0f-49d8-a23b-53dd6d666355-kube-api-access-whcpz" pod "e7947c08-1f0f-49d8-a23b-53dd6d666355" (UID: "e7947c08-1f0f-49d8-a23b-53dd6d666355"). InnerVolumeSpecName "kube-api-access-whcpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:13.812853 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.812815    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"]
Nov 03 19:44:13.812984 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:13.812961    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode72488d0_dae8_4b0f_9c8c_9695319069b5.slice/crio-bcdeef383da8593a0765fbf6345f3a1bdfe1af2be87abc0772a68823f0b19b4a WatchSource:0}: Error finding container bcdeef383da8593a0765fbf6345f3a1bdfe1af2be87abc0772a68823f0b19b4a: Status 404 returned error can't find the container with id bcdeef383da8593a0765fbf6345f3a1bdfe1af2be87abc0772a68823f0b19b4a
Nov 03 19:44:13.910780 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.910752    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:13.910780 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.910777    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whcpz\" (UniqueName: \"kubernetes.io/projected/e7947c08-1f0f-49d8-a23b-53dd6d666355-kube-api-access-whcpz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:13.910970 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:13.910788    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7947c08-1f0f-49d8-a23b-53dd6d666355-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:14.244147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:14.239989    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:14.608006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:14.607909    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" event={"ID":"e72488d0-dae8-4b0f-9c8c-9695319069b5","Type":"ContainerStarted","Data":"42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082"}
Nov 03 19:44:14.608006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:14.607965    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" event={"ID":"e72488d0-dae8-4b0f-9c8c-9695319069b5","Type":"ContainerStarted","Data":"bcdeef383da8593a0765fbf6345f3a1bdfe1af2be87abc0772a68823f0b19b4a"}
Nov 03 19:44:14.609386 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:14.609366    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"
Nov 03 19:44:14.609460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:14.609384    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t" event={"ID":"e7947c08-1f0f-49d8-a23b-53dd6d666355","Type":"ContainerDied","Data":"5672e14bbf54be78a6fc806b087318417c1f0333999617341c28c7b08c6a6572"}
Nov 03 19:44:14.609460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:14.609425    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5672e14bbf54be78a6fc806b087318417c1f0333999617341c28c7b08c6a6572"
Nov 03 19:44:14.625823 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:14.625779    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podStartSLOduration=1.6257665129999999 podStartE2EDuration="1.625766513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:14.621770287 +0000 UTC m=+767.891893581" watchObservedRunningTime="2026-11-03 19:44:14.625766513 +0000 UTC m=+767.895889808"
Nov 03 19:44:15.059460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:15.059144    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:15.294220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:15.294153    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:15.534101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:15.534063    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:15.864545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:15.858506    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:16.608834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.608804    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"]
Nov 03 19:44:16.609365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.609351    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e7947c08-1f0f-49d8-a23b-53dd6d666355" containerName="revoke-keys"
Nov 03 19:44:16.609420 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.609366    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7947c08-1f0f-49d8-a23b-53dd6d666355" containerName="revoke-keys"
Nov 03 19:44:16.609475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.609466    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7947c08-1f0f-49d8-a23b-53dd6d666355" containerName="revoke-keys"
Nov 03 19:44:16.613132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.613111    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:16.620385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.618852    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:44:16.620385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.619475    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:44:16.622497 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.622457    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"]
Nov 03 19:44:16.642554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.642525    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfa4f37b-fgnbx\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:16.642683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.642558    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfa4f37b-fgnbx\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:16.642742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.642703    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmp98\" (UniqueName: \"kubernetes.io/projected/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-kube-api-access-vmp98\") pod \"maas-api-revoke-sub-cfa4f37b-fgnbx\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:16.743690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.743653    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfa4f37b-fgnbx\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:16.743690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.743693    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfa4f37b-fgnbx\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:16.743900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.743757    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmp98\" (UniqueName: \"kubernetes.io/projected/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-kube-api-access-vmp98\") pod \"maas-api-revoke-sub-cfa4f37b-fgnbx\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:16.744397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.744370    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfa4f37b-fgnbx\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:16.746204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.746184    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfa4f37b-fgnbx\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:16.753615 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.753590    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmp98\" (UniqueName: \"kubernetes.io/projected/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-kube-api-access-vmp98\") pod \"maas-api-revoke-sub-cfa4f37b-fgnbx\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:16.794094 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.794064    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:16.934410 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:16.934373    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:17.016240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.013524    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"]
Nov 03 19:44:17.022186 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.022139    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:17.051348 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.051259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3675c37f-vssgq\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:17.052347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.052319    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3675c37f-vssgq\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:17.052675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.052650    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5d8\" (UniqueName: \"kubernetes.io/projected/1081e45b-8763-4e99-acb3-b94a192fa8a8-kube-api-access-rp5d8\") pod \"maas-api-revoke-sub-3675c37f-vssgq\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:17.241368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.241273    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3675c37f-vssgq\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:17.241666 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.241642    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5d8\" (UniqueName: \"kubernetes.io/projected/1081e45b-8763-4e99-acb3-b94a192fa8a8-kube-api-access-rp5d8\") pod \"maas-api-revoke-sub-3675c37f-vssgq\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:17.254405 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.241819    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3675c37f-vssgq\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:17.261136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.250304    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3675c37f-vssgq\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:17.285441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.270211    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"]
Nov 03 19:44:17.417197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.406139    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3675c37f-vssgq\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:17.509947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.454195    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5d8\" (UniqueName: \"kubernetes.io/projected/1081e45b-8763-4e99-acb3-b94a192fa8a8-kube-api-access-rp5d8\") pod \"maas-api-revoke-sub-3675c37f-vssgq\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:17.509947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.459612    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:17.562554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.562247    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:17.589486 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:17.589447    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc6006b_a4d1_4f1d_b761_ad0fdc5a52e6.slice/crio-47b20944d1e2efd69a4a4cb86b91c85fb72778657682508e4cd641a0cf104bf5 WatchSource:0}: Error finding container 47b20944d1e2efd69a4a4cb86b91c85fb72778657682508e4cd641a0cf104bf5: Status 404 returned error can't find the container with id 47b20944d1e2efd69a4a4cb86b91c85fb72778657682508e4cd641a0cf104bf5
Nov 03 19:44:17.594293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.592258    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"]
Nov 03 19:44:17.752415 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.752313    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx" event={"ID":"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6","Type":"ContainerStarted","Data":"47b20944d1e2efd69a4a4cb86b91c85fb72778657682508e4cd641a0cf104bf5"}
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.957631    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"]
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:17.981476    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.049503    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"]
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.053494    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.053533    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85beffbc-e581-4e37-b139-68118e87212b-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.053613    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.053640    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.053693    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.053746    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85beffbc-e581-4e37-b139-68118e87212b-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.053791    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gzpc\" (UniqueName: \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-kube-api-access-4gzpc\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.053966    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.098340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.054069    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.159690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.159475    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.159690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.159548    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.159690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.159577    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85beffbc-e581-4e37-b139-68118e87212b-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.159690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.159665    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.159926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.159696    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.159926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.159760    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.159926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.159818    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85beffbc-e581-4e37-b139-68118e87212b-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.159926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.159856    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gzpc\" (UniqueName: \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-kube-api-access-4gzpc\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.159926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.159899    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.182229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.160550    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.182229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.161526    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.182229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.161849    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.182229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.162686    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85beffbc-e581-4e37-b139-68118e87212b-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.182229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.163006    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.215863 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.215787    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.218868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.218013    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gzpc\" (UniqueName: \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-kube-api-access-4gzpc\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.218868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.218408    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.218868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.218821    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85beffbc-e581-4e37-b139-68118e87212b-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.476333 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.476299    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:18.540522 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.540382    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:18.588690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.588591    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"]
Nov 03 19:44:18.601255 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:18.601207    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1081e45b_8763_4e99_acb3_b94a192fa8a8.slice/crio-89533c6a4ba80ab5cee2b018ae72433dc18707b1df28811690ae26a4a0e3dbe7 WatchSource:0}: Error finding container 89533c6a4ba80ab5cee2b018ae72433dc18707b1df28811690ae26a4a0e3dbe7: Status 404 returned error can't find the container with id 89533c6a4ba80ab5cee2b018ae72433dc18707b1df28811690ae26a4a0e3dbe7
Nov 03 19:44:18.764379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.764340    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq" event={"ID":"1081e45b-8763-4e99-acb3-b94a192fa8a8","Type":"ContainerStarted","Data":"89533c6a4ba80ab5cee2b018ae72433dc18707b1df28811690ae26a4a0e3dbe7"}
Nov 03 19:44:18.766145 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.766120    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx" event={"ID":"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6","Type":"ContainerStarted","Data":"bfd2cb980961a49b8910914b5a5997c86a33eb7bcc8c04f0432cd5c9e30eaab5"}
Nov 03 19:44:18.782126 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.782071    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx" podStartSLOduration=2.782058609 podStartE2EDuration="2.782058609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:18.77875033 +0000 UTC m=+772.048873625" watchObservedRunningTime="2026-11-03 19:44:18.782058609 +0000 UTC m=+772.052181904"
Nov 03 19:44:18.924029 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:18.923994    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85beffbc_e581_4e37_b139_68118e87212b.slice/crio-47b4e34a4cba4f360f59e4bff1bfcb748a0342a29dd9db7bdb43ea6f556fe336 WatchSource:0}: Error finding container 47b4e34a4cba4f360f59e4bff1bfcb748a0342a29dd9db7bdb43ea6f556fe336: Status 404 returned error can't find the container with id 47b4e34a4cba4f360f59e4bff1bfcb748a0342a29dd9db7bdb43ea6f556fe336
Nov 03 19:44:18.927234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.925018    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"]
Nov 03 19:44:18.931719 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.931684    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:18.931791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.931753    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:18.931791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:18.931783    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:19.374932 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:19.372716    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:19.774067 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:19.774033    4531 generic.go:356] "Generic (PLEG): container finished" podID="1081e45b-8763-4e99-acb3-b94a192fa8a8" containerID="9d7895667addc8ef631538583dd021aa5649601439da0e557af00a98dd906209" exitCode=0
Nov 03 19:44:19.774556 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:19.774096    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq" event={"ID":"1081e45b-8763-4e99-acb3-b94a192fa8a8","Type":"ContainerDied","Data":"9d7895667addc8ef631538583dd021aa5649601439da0e557af00a98dd906209"}
Nov 03 19:44:19.775841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:19.775814    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" event={"ID":"85beffbc-e581-4e37-b139-68118e87212b","Type":"ContainerStarted","Data":"5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0"}
Nov 03 19:44:19.775976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:19.775862    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" event={"ID":"85beffbc-e581-4e37-b139-68118e87212b","Type":"ContainerStarted","Data":"47b4e34a4cba4f360f59e4bff1bfcb748a0342a29dd9db7bdb43ea6f556fe336"}
Nov 03 19:44:19.777580 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:19.777557    4531 generic.go:356] "Generic (PLEG): container finished" podID="3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6" containerID="bfd2cb980961a49b8910914b5a5997c86a33eb7bcc8c04f0432cd5c9e30eaab5" exitCode=0
Nov 03 19:44:19.777689 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:19.777622    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx" event={"ID":"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6","Type":"ContainerDied","Data":"bfd2cb980961a49b8910914b5a5997c86a33eb7bcc8c04f0432cd5c9e30eaab5"}
Nov 03 19:44:19.818774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:19.818722    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" podStartSLOduration=2.818709576 podStartE2EDuration="2.818709576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:19.816762711 +0000 UTC m=+773.086886008" watchObservedRunningTime="2026-11-03 19:44:19.818709576 +0000 UTC m=+773.088832870"
Nov 03 19:44:20.150199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:20.146905    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:20.477410 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:20.477319    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:20.478835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:20.478806    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:44:20.478971 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:20.478861    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:44:20.588932 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:44:20.588894    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:44:21.031370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.028381    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:21.093797 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.093773    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:21.100001 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.099981    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:21.229000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.228968    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-xks-ca\") pod \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") "
Nov 03 19:44:21.229198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.229032    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-openshift-ca\") pod \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") "
Nov 03 19:44:21.229198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.229099    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-openshift-ca\") pod \"1081e45b-8763-4e99-acb3-b94a192fa8a8\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") "
Nov 03 19:44:21.229198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.229128    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1081e45b-8763-4e99-acb3-b94a192fa8a8-kube-api-access-rp5d8\" (UniqueName: \"kubernetes.io/projected/1081e45b-8763-4e99-acb3-b94a192fa8a8-kube-api-access-rp5d8\") pod \"1081e45b-8763-4e99-acb3-b94a192fa8a8\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") "
Nov 03 19:44:21.229364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.229229    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-xks-ca\") pod \"1081e45b-8763-4e99-acb3-b94a192fa8a8\" (UID: \"1081e45b-8763-4e99-acb3-b94a192fa8a8\") "
Nov 03 19:44:21.229364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.229281    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-kube-api-access-vmp98\" (UniqueName: \"kubernetes.io/projected/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-kube-api-access-vmp98\") pod \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\" (UID: \"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6\") "
Nov 03 19:44:21.229478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.229447    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-openshift-ca" pod "3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6" (UID: "3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:21.229554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.229524    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-openshift-ca" pod "1081e45b-8763-4e99-acb3-b94a192fa8a8" (UID: "1081e45b-8763-4e99-acb3-b94a192fa8a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:21.229801 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.229776    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.229801 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.229800    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.231168 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.231141    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-xks-ca" pod "3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6" (UID: "3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:21.231455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.231435    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-kube-api-access-vmp98" pod "3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6" (UID: "3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6"). InnerVolumeSpecName "kube-api-access-vmp98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:21.231713 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.231693    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-xks-ca" pod "1081e45b-8763-4e99-acb3-b94a192fa8a8" (UID: "1081e45b-8763-4e99-acb3-b94a192fa8a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:21.231713 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.231706    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1081e45b-8763-4e99-acb3-b94a192fa8a8-kube-api-access-rp5d8" pod "1081e45b-8763-4e99-acb3-b94a192fa8a8" (UID: "1081e45b-8763-4e99-acb3-b94a192fa8a8"). InnerVolumeSpecName "kube-api-access-rp5d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:21.330533 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.330499    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1081e45b-8763-4e99-acb3-b94a192fa8a8-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.330533 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.330529    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmp98\" (UniqueName: \"kubernetes.io/projected/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-kube-api-access-vmp98\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.330533 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.330541    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.330746 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.330550    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp5d8\" (UniqueName: \"kubernetes.io/projected/1081e45b-8763-4e99-acb3-b94a192fa8a8-kube-api-access-rp5d8\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.477707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.477663    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:44:21.477904 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.477729    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:44:21.792784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.792707    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"
Nov 03 19:44:21.792784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.792752    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq" event={"ID":"1081e45b-8763-4e99-acb3-b94a192fa8a8","Type":"ContainerDied","Data":"89533c6a4ba80ab5cee2b018ae72433dc18707b1df28811690ae26a4a0e3dbe7"}
Nov 03 19:44:21.792990 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.792819    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89533c6a4ba80ab5cee2b018ae72433dc18707b1df28811690ae26a4a0e3dbe7"
Nov 03 19:44:21.794387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.794359    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"
Nov 03 19:44:21.794387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.794374    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx" event={"ID":"3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6","Type":"ContainerDied","Data":"47b20944d1e2efd69a4a4cb86b91c85fb72778657682508e4cd641a0cf104bf5"}
Nov 03 19:44:21.794577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.794432    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47b20944d1e2efd69a4a4cb86b91c85fb72778657682508e4cd641a0cf104bf5"
Nov 03 19:44:21.864794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:21.864709    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:22.477858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:22.477819    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:44:22.478327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:22.477876    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:44:22.875257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:22.870718    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:23.477529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:23.477487    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:44:23.477718 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:23.477575    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:44:23.533943 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:23.533905    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:23.565877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:23.565552    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:23.571557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:23.571531    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:23.818184 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:23.818090    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:44:24.004197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:23.992460    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:24.477786 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:24.477747    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:44:24.477981 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:24.477808    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:44:24.656199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:24.650049    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:24.840248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:24.836256    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:25.180679 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.180635    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"]
Nov 03 19:44:25.181228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.181210    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6" containerName="revoke-keys"
Nov 03 19:44:25.181228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.181226    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6" containerName="revoke-keys"
Nov 03 19:44:25.181418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.181242    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1081e45b-8763-4e99-acb3-b94a192fa8a8" containerName="revoke-keys"
Nov 03 19:44:25.181418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.181248    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1081e45b-8763-4e99-acb3-b94a192fa8a8" containerName="revoke-keys"
Nov 03 19:44:25.181418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.181356    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6" containerName="revoke-keys"
Nov 03 19:44:25.181418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.181376    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1081e45b-8763-4e99-acb3-b94a192fa8a8" containerName="revoke-keys"
Nov 03 19:44:25.186319 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.186288    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:25.192508 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.192484    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:44:25.192839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.192816    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"]
Nov 03 19:44:25.193139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.193113    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:44:25.210510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.210483    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcsfk\" (UniqueName: \"kubernetes.io/projected/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-kube-api-access-kcsfk\") pod \"maas-api-revoke-sub-df208d50-wg264\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:25.210644 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.210629    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df208d50-wg264\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:25.210729 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.210694    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df208d50-wg264\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:25.312283 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.312244    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcsfk\" (UniqueName: \"kubernetes.io/projected/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-kube-api-access-kcsfk\") pod \"maas-api-revoke-sub-df208d50-wg264\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:25.312483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.312394    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df208d50-wg264\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:25.312483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.312432    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df208d50-wg264\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:25.313243 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.313220    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df208d50-wg264\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:25.315012 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.314986    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df208d50-wg264\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:25.320358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.320335    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcsfk\" (UniqueName: \"kubernetes.io/projected/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-kube-api-access-kcsfk\") pod \"maas-api-revoke-sub-df208d50-wg264\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:25.477327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.477237    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:44:25.477474 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.477350    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:44:25.512600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.509416    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:25.684228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.674963    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:25.837620 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:25.837594    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"]
Nov 03 19:44:25.839330 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:25.839294    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c35dcf_fe30_4e9b_86fd_ca5d0a7cf229.slice/crio-5fa44a16fc338841f20e21b21c151dc19070caa85eac4631fab6f04d0bfa15df WatchSource:0}: Error finding container 5fa44a16fc338841f20e21b21c151dc19070caa85eac4631fab6f04d0bfa15df: Status 404 returned error can't find the container with id 5fa44a16fc338841f20e21b21c151dc19070caa85eac4631fab6f04d0bfa15df
Nov 03 19:44:26.413243 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:26.394374    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:26.481630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:26.481602    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:26.482144 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:26.482113    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:26.482750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:26.482734    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 19:44:26.831115 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:26.831084    4531 generic.go:356] "Generic (PLEG): container finished" podID="53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229" containerID="e58b3b5069a1632d5722bad0d501dbbdea75f86049f8fbe76827b57178ad7a86" exitCode=0
Nov 03 19:44:26.831599 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:26.831162    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264" event={"ID":"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229","Type":"ContainerDied","Data":"e58b3b5069a1632d5722bad0d501dbbdea75f86049f8fbe76827b57178ad7a86"}
Nov 03 19:44:26.831599 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:26.831244    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264" event={"ID":"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229","Type":"ContainerStarted","Data":"5fa44a16fc338841f20e21b21c151dc19070caa85eac4631fab6f04d0bfa15df"}
Nov 03 19:44:27.124895 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:27.124805    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:27.889675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:27.889636    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:28.027974 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.027948    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:28.055070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.055040    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-xks-ca\") pod \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") "
Nov 03 19:44:28.055220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.055112    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-openshift-ca\") pod \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") "
Nov 03 19:44:28.055295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.055217    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-kube-api-access-kcsfk\" (UniqueName: \"kubernetes.io/projected/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-kube-api-access-kcsfk\") pod \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\" (UID: \"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229\") "
Nov 03 19:44:28.055554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.055522    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-openshift-ca" pod "53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229" (UID: "53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:28.055754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.055733    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:28.057426 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.057391    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-kube-api-access-kcsfk" pod "53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229" (UID: "53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229"). InnerVolumeSpecName "kube-api-access-kcsfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:28.057612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.057584    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-xks-ca" pod "53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229" (UID: "53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:28.157008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.156979    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:28.157008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.157008    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcsfk\" (UniqueName: \"kubernetes.io/projected/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229-kube-api-access-kcsfk\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:28.767207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.764973    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:28.845862 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.845828    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264" event={"ID":"53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229","Type":"ContainerDied","Data":"5fa44a16fc338841f20e21b21c151dc19070caa85eac4631fab6f04d0bfa15df"}
Nov 03 19:44:28.846013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.845877    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fa44a16fc338841f20e21b21c151dc19070caa85eac4631fab6f04d0bfa15df"
Nov 03 19:44:28.846013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:28.845886    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"
Nov 03 19:44:29.574148 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:29.574104    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:30.232194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:30.231949    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:30.961619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:30.961554    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:31.647209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:31.647166    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:32.274271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.274235    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"]
Nov 03 19:44:32.295220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.275771    4531 cpu_manager.go:414] "containerMap: removing container" podUID="53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229" containerName="revoke-keys"
Nov 03 19:44:32.297492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.297453    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229" containerName="revoke-keys"
Nov 03 19:44:32.297913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.297893    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229" containerName="revoke-keys"
Nov 03 19:44:32.311392 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.297276    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:32.324389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.324357    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.333510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.332863    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-82dbbf\"/\"distinct-w0-82dbbf-kserve-self-signed-certs\""
Nov 03 19:44:32.408041 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.408009    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"]
Nov 03 19:44:32.438640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.437669    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-tmp-dir\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.438640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.437737    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8306ce04-6749-4236-9359-895705d5fbe1-tls-certs\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.438640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.437863    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-home\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.438640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.437927    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdck\" (UniqueName: \"kubernetes.io/projected/8306ce04-6749-4236-9359-895705d5fbe1-kube-api-access-vbdck\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.438640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.437965    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-model-cache\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.438640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.438099    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-dshm\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.544090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.542502    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-model-cache\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.544090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.542646    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-dshm\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.544090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.542704    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-tmp-dir\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.544090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.542752    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8306ce04-6749-4236-9359-895705d5fbe1-tls-certs\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.544090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.542876    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-home\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.544090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.542940    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdck\" (UniqueName: \"kubernetes.io/projected/8306ce04-6749-4236-9359-895705d5fbe1-kube-api-access-vbdck\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.544090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.543707    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-model-cache\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.546935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.546605    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-tmp-dir\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.546935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.546867    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-home\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.598212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.574220    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"]
Nov 03 19:44:32.642195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.637123    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.645968 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.645885    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8306ce04-6749-4236-9359-895705d5fbe1-tls-certs\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.646510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.646453    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-dshm\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.714092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.713857    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"]
Nov 03 19:44:32.754292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.752806    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.754292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.752862    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.754292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.752925    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.754292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.752955    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.754292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.752979    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2rms\" (UniqueName: \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-kube-api-access-s2rms\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.754292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.753037    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.754292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.753190    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.754292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.753223    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.754292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.753261    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a353656-7621-47f3-a298-ef89b4a6ced9-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.760213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.759774    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdck\" (UniqueName: \"kubernetes.io/projected/8306ce04-6749-4236-9359-895705d5fbe1-kube-api-access-vbdck\") pod \"distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:32.870214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.856940    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.870214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.857003    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.870214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.857065    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.870214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.857097    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.870214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.857125    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2rms\" (UniqueName: \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-kube-api-access-s2rms\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.870214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.857209    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.870214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.857352    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.870214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.857381    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.870214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.857421    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a353656-7621-47f3-a298-ef89b4a6ced9-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.870214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.858446    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a353656-7621-47f3-a298-ef89b4a6ced9-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.880431 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.880387    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.886626 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.886585    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.887195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.887146    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:32.921241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:32.900773    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:33.184199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:33.150850    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:33.184440 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:33.154968    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2rms\" (UniqueName: \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-kube-api-access-s2rms\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:33.184565 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:33.155248    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:33.205216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:33.166572    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:33.205410 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:33.170839    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:33.491248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:33.480427    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:33.692259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:33.692167    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"]
Nov 03 19:44:33.931673 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:33.931640    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"]
Nov 03 19:44:33.931869 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:33.931842    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a353656_7621_47f3_a298_ef89b4a6ced9.slice/crio-396447d60a723baff08249d4def8735ddcd8e15e4567171e5efc67ab2ff59b63 WatchSource:0}: Error finding container 396447d60a723baff08249d4def8735ddcd8e15e4567171e5efc67ab2ff59b63: Status 404 returned error can't find the container with id 396447d60a723baff08249d4def8735ddcd8e15e4567171e5efc67ab2ff59b63
Nov 03 19:44:33.934357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:33.934327    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:33.934423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:33.934389    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:33.934423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:33.934417    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:34.217419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.217381    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" event={"ID":"8306ce04-6749-4236-9359-895705d5fbe1","Type":"ContainerStarted","Data":"47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528"}
Nov 03 19:44:34.217685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.217639    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" event={"ID":"8306ce04-6749-4236-9359-895705d5fbe1","Type":"ContainerStarted","Data":"9fa1e7587aee6351a04b45ca9357842230dc4f87c3733fc96991a17359f02e9a"}
Nov 03 19:44:34.222596 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.222566    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" event={"ID":"5a353656-7621-47f3-a298-ef89b4a6ced9","Type":"ContainerStarted","Data":"9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708"}
Nov 03 19:44:34.222733 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.222623    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" event={"ID":"5a353656-7621-47f3-a298-ef89b4a6ced9","Type":"ContainerStarted","Data":"396447d60a723baff08249d4def8735ddcd8e15e4567171e5efc67ab2ff59b63"}
Nov 03 19:44:34.233437 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.233394    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podStartSLOduration=2.233381779 podStartE2EDuration="2.233381779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:34.230896117 +0000 UTC m=+787.501019403" watchObservedRunningTime="2026-11-03 19:44:34.233381779 +0000 UTC m=+787.503505074"
Nov 03 19:44:34.254916 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.254843    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" podStartSLOduration=2.254824332 podStartE2EDuration="2.254824332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:34.250662627 +0000 UTC m=+787.520785923" watchObservedRunningTime="2026-11-03 19:44:34.254824332 +0000 UTC m=+787.524947627"
Nov 03 19:44:34.286911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.286877    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:34.378359 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.377879    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"]
Nov 03 19:44:34.384477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.384451    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.392108 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.392072    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"]
Nov 03 19:44:34.393248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.393223    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f4e123\"/\"distinct-w4-f4e123-kserve-self-signed-certs\""
Nov 03 19:44:34.437447 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.437385    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-home\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.437645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.437459    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-model-cache\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.437645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.437581    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8zwc\" (UniqueName: \"kubernetes.io/projected/32c9fcac-59af-4faa-a5ae-0d78688a820b-kube-api-access-v8zwc\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.437754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.437659    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-dshm\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.437754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.437691    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-tmp-dir\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.437862 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.437762    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c9fcac-59af-4faa-a5ae-0d78688a820b-tls-certs\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.465055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.465013    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:34.503118 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.503034    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:34.505228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.505200    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:44:34.505452 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.505425    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:44:34.548198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.541426    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-home\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.548198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.541505    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-model-cache\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.548198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.541677    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8zwc\" (UniqueName: \"kubernetes.io/projected/32c9fcac-59af-4faa-a5ae-0d78688a820b-kube-api-access-v8zwc\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.548198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.541807    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-dshm\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.548198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.541847    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-tmp-dir\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.548198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.541995    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c9fcac-59af-4faa-a5ae-0d78688a820b-tls-certs\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.572728 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.571549    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-model-cache\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.573281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.573254    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-home\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.574122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.574062    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-tmp-dir\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.600003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.599921    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8zwc\" (UniqueName: \"kubernetes.io/projected/32c9fcac-59af-4faa-a5ae-0d78688a820b-kube-api-access-v8zwc\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.600003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.599955    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-dshm\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.610143 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.610112    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c9fcac-59af-4faa-a5ae-0d78688a820b-tls-certs\") pod \"distinct-w4-f4e123-kserve-6d768c548b-ptfsz\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:34.708389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:34.708355    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:35.000434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:35.000408    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"]
Nov 03 19:44:35.002254 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:35.002227    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32c9fcac_59af_4faa_a5ae_0d78688a820b.slice/crio-40fc58f5732278f4fc61a4c9de64c87e5332df0c7c5b12cc7ccc43b4ab2ba3c0 WatchSource:0}: Error finding container 40fc58f5732278f4fc61a4c9de64c87e5332df0c7c5b12cc7ccc43b4ab2ba3c0: Status 404 returned error can't find the container with id 40fc58f5732278f4fc61a4c9de64c87e5332df0c7c5b12cc7ccc43b4ab2ba3c0
Nov 03 19:44:35.168308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:35.168273    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:35.231093 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:35.231055    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" event={"ID":"32c9fcac-59af-4faa-a5ae-0d78688a820b","Type":"ContainerStarted","Data":"1592c95b3a604b9241ab73b1405d3ff745ef6025436a5e04f51d33b30fce89f2"}
Nov 03 19:44:35.231310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:35.231104    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" event={"ID":"32c9fcac-59af-4faa-a5ae-0d78688a820b","Type":"ContainerStarted","Data":"40fc58f5732278f4fc61a4c9de64c87e5332df0c7c5b12cc7ccc43b4ab2ba3c0"}
Nov 03 19:44:35.264202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:35.262217    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podStartSLOduration=1.2621964110000001 podStartE2EDuration="1.262196411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:35.260447974 +0000 UTC m=+788.530571270" watchObservedRunningTime="2026-11-03 19:44:35.262196411 +0000 UTC m=+788.532319707"
Nov 03 19:44:35.264202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:35.263122    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:35.502592 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:35.502552    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:44:35.502797 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:35.502611    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:44:35.708639 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:35.708599    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:36.167737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:36.167698    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:36.503026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:36.502945    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:44:36.503026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:36.502996    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:44:37.159193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:37.154676    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:37.503143 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:37.503061    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:44:37.503143 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:37.503114    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:44:38.000691 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:38.000653    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:38.503465 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:38.503424    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:44:38.503864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:38.503498    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:44:38.861937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:38.861838    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:39.502722 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:39.502685    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:44:39.502914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:39.502765    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:44:39.869884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:39.861135    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:40.503430 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:40.503395    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 19:44:40.503602 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:40.503451    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 19:44:40.734200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:40.731897    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:41.440194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:41.439406    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:41.507200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:41.507154    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:41.507689 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:41.507668    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:41.508435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:41.508418    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:44:42.140377 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:42.140344    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:42.348275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:42.348235    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:43.167915 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.167878    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:43.204124 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.204085    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:43.230794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.230760    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:43.256004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.255973    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"]
Nov 03 19:44:43.264746 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.264717    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.265600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.265567    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"]
Nov 03 19:44:43.268795 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.268770    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0486e8\"/\"distinct-2-w5-0486e8-kserve-self-signed-certs\""
Nov 03 19:44:43.302227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.302197    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0fc563f-e643-452b-91bd-fee9d1e8e739-tls-certs\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.302385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.302368    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-tmp-dir\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.302460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.302412    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqmbx\" (UniqueName: \"kubernetes.io/projected/c0fc563f-e643-452b-91bd-fee9d1e8e739-kube-api-access-sqmbx\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.302460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.302431    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-model-cache\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.302577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.302537    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-home\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.302577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.302572    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-dshm\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.325629 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.325601    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:44:43.361479 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.361445    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:43.404367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.404333    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-tmp-dir\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.404545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.404408    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqmbx\" (UniqueName: \"kubernetes.io/projected/c0fc563f-e643-452b-91bd-fee9d1e8e739-kube-api-access-sqmbx\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.404545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.404439    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-model-cache\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.404545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.404527    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-home\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.404748 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.404580    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-dshm\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.404808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.404745    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0fc563f-e643-452b-91bd-fee9d1e8e739-tls-certs\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.404894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.404868    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-tmp-dir\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.404966 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.404945    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-model-cache\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.405058 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.405044    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-home\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.406883 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.406865    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-dshm\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.407083 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.407065    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0fc563f-e643-452b-91bd-fee9d1e8e739-tls-certs\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.411757 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.411734    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqmbx\" (UniqueName: \"kubernetes.io/projected/c0fc563f-e643-452b-91bd-fee9d1e8e739-kube-api-access-sqmbx\") pod \"distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") " pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.584114 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.584035    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:43.861166 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:43.861136    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0fc563f_e643_452b_91bd_fee9d1e8e739.slice/crio-026680fcec3683cdf8fd4fe4afdf88e72cc9ea30026008a715394b65471a08ef WatchSource:0}: Error finding container 026680fcec3683cdf8fd4fe4afdf88e72cc9ea30026008a715394b65471a08ef: Status 404 returned error can't find the container with id 026680fcec3683cdf8fd4fe4afdf88e72cc9ea30026008a715394b65471a08ef
Nov 03 19:44:43.861316 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:43.861230    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"]
Nov 03 19:44:44.032544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:44.022933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:44.325315 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:44.325275    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" event={"ID":"c0fc563f-e643-452b-91bd-fee9d1e8e739","Type":"ContainerStarted","Data":"34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2"}
Nov 03 19:44:44.325759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:44.325327    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" event={"ID":"c0fc563f-e643-452b-91bd-fee9d1e8e739","Type":"ContainerStarted","Data":"026680fcec3683cdf8fd4fe4afdf88e72cc9ea30026008a715394b65471a08ef"}
Nov 03 19:44:44.341366 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:44.341317    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podStartSLOduration=1.34130347 podStartE2EDuration="1.34130347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:44.339645823 +0000 UTC m=+797.609769119" watchObservedRunningTime="2026-11-03 19:44:44.34130347 +0000 UTC m=+797.611426764"
Nov 03 19:44:44.584219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:44.584114    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:44.708995 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:44.708968    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:44.721733 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:44.721708    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:44.736470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:44.736438    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:45.424510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:45.424482    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:44:45.700773 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:45.700679    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:47.199275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.199236    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:47.315707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.315665    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:47.359997 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.359942    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:47.498756 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.498663    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"]
Nov 03 19:44:47.504708 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.504676    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.516042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.515715    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"]
Nov 03 19:44:47.568320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.568270    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.568320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.568322    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.568563 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.568373    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.568563 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.568399    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2lwf\" (UniqueName: \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-kube-api-access-q2lwf\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.568563 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.568432    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.568563 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.568476    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.568563 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.568529    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.568807 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.568652    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.568807 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.568756    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.623473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.623434    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"]
Nov 03 19:44:47.627608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.627588    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.642330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.642302    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"]
Nov 03 19:44:47.670632 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.670596    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.670843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.670678    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.670843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.670742    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.670843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.670757    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.670843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.670797    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.670843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.670823    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2lwf\" (UniqueName: \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-kube-api-access-q2lwf\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.671138 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.670855    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.671138 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.670900    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.671138 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.670949    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.671343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.671232    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.671343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.671270    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.671641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.671612    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.671941 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.671916    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.672029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.671917    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.674400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.674379    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.674781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.674760    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.685513 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.685486    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2lwf\" (UniqueName: \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-kube-api-access-q2lwf\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.685625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.685603    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.772796 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.772711    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.772796 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.772770    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.772975 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.772910    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.772975 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.772936    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.772975 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.772960    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.773088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.773069    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fspjf\" (UniqueName: \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-kube-api-access-fspjf\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.773146 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.773135    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.773222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.773207    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.773277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.773263    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.820868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.820830    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:47.830899 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.830877    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:47.880204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.879091    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.880204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.879165    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.880204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.879276    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.880204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.879300    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.880204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.879329    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.880204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.879391    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fspjf\" (UniqueName: \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-kube-api-access-fspjf\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.880204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.879474    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.880204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.879524    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.880204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.879571    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.919195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.905099    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.919195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.915713    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.919195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.915935    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.941981 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.929773    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.941981 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.930527    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.941981 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.931683    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.941981 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.932927    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.941981 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.934371    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:47.947850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:47.947015    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fspjf\" (UniqueName: \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-kube-api-access-fspjf\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:48.067718 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.067628    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"]
Nov 03 19:44:48.073884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.073864    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.080739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.080681    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"]
Nov 03 19:44:48.153754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.153716    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:48.183947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.183911    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.184115 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.183966    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.184115 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.184092    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.184214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.184123    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq6q4\" (UniqueName: \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-kube-api-access-mq6q4\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.184281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.184268    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.184372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.184359    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.184409 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.184391    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.184486 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.184459    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.184530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.184514    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.199450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.199427    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"]
Nov 03 19:44:48.199828 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:48.199805    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17152ed_5f0b_4ef6_abb6_ff06e7a449e8.slice/crio-78468aab5fb91158161925aeed2897d585da33b2df5e866f9208cc7c67e3f154 WatchSource:0}: Error finding container 78468aab5fb91158161925aeed2897d585da33b2df5e866f9208cc7c67e3f154: Status 404 returned error can't find the container with id 78468aab5fb91158161925aeed2897d585da33b2df5e866f9208cc7c67e3f154
Nov 03 19:44:48.202325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.202294    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:48.202400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.202355    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:48.202400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.202387    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:48.245494 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.245468    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:48.285385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.285353    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.285556 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.285438    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.285556 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.285462    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.285556 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.285491    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.285556 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.285517    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.285793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.285593    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.285793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.285630    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.285793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.285693    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.285793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.285720    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq6q4\" (UniqueName: \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-kube-api-access-mq6q4\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.285992 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.285867    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.286201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.286140    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.286435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.286414    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.286507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.286469    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.286725 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.286705    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.289158 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.288401    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.290289 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.290264    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.292985 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.292960    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq6q4\" (UniqueName: \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-kube-api-access-mq6q4\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.293317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.293298    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.391433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.391400    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:48.437882 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.437846    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" event={"ID":"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8","Type":"ContainerStarted","Data":"6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83"}
Nov 03 19:44:48.438067 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.437903    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" event={"ID":"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8","Type":"ContainerStarted","Data":"78468aab5fb91158161925aeed2897d585da33b2df5e866f9208cc7c67e3f154"}
Nov 03 19:44:48.459474 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.459416    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" podStartSLOduration=1.459396871 podStartE2EDuration="1.459396871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:48.453455006 +0000 UTC m=+801.723578301" watchObservedRunningTime="2026-11-03 19:44:48.459396871 +0000 UTC m=+801.729520168"
Nov 03 19:44:48.530994 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.530821    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"]
Nov 03 19:44:48.538004 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:48.537971    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896c41d2_a2bf_44a6_abdd_92c3d2ac0a04.slice/crio-14f03f7f6461fdd3fb8e385a806370ca47ad42a6ed5d20f33817e3080f5ec2e7 WatchSource:0}: Error finding container 14f03f7f6461fdd3fb8e385a806370ca47ad42a6ed5d20f33817e3080f5ec2e7: Status 404 returned error can't find the container with id 14f03f7f6461fdd3fb8e385a806370ca47ad42a6ed5d20f33817e3080f5ec2e7
Nov 03 19:44:48.541442 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.541394    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:48.541535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.541484    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:48.541575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.541530    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:48.684502 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:48.684469    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa36d9d_eeaf_4a63_99da_862d2b923b0f.slice/crio-4c404812a5a231a76d7fa7365dde64c9580d9b24f1eaf618784d0aa96dcee195 WatchSource:0}: Error finding container 4c404812a5a231a76d7fa7365dde64c9580d9b24f1eaf618784d0aa96dcee195: Status 404 returned error can't find the container with id 4c404812a5a231a76d7fa7365dde64c9580d9b24f1eaf618784d0aa96dcee195
Nov 03 19:44:48.686061 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.686030    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"]
Nov 03 19:44:48.687822 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.687786    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:48.687910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.687868    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:48.687946 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.687909    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:48.831313 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.831275    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:48.934932 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:48.932987    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:49.491249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:49.482688    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" event={"ID":"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04","Type":"ContainerStarted","Data":"bb710b5d10d4db0b0711e14186983e0cb5fa2d6a7b089a2bc777ec9487f8f643"}
Nov 03 19:44:49.491249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:49.482764    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" event={"ID":"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04","Type":"ContainerStarted","Data":"14f03f7f6461fdd3fb8e385a806370ca47ad42a6ed5d20f33817e3080f5ec2e7"}
Nov 03 19:44:49.502959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:49.502922    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" event={"ID":"eaa36d9d-eeaf-4a63-99da-862d2b923b0f","Type":"ContainerStarted","Data":"d5755f40c9b00101a1b050a2c8096b4df71b0a0ea4726d0cb606f14f45fc28d3"}
Nov 03 19:44:49.503392 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:49.503368    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" event={"ID":"eaa36d9d-eeaf-4a63-99da-862d2b923b0f","Type":"ContainerStarted","Data":"4c404812a5a231a76d7fa7365dde64c9580d9b24f1eaf618784d0aa96dcee195"}
Nov 03 19:44:49.520481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:49.520428    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" podStartSLOduration=2.52041308 podStartE2EDuration="2.52041308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:49.512517082 +0000 UTC m=+802.782640378" watchObservedRunningTime="2026-11-03 19:44:49.52041308 +0000 UTC m=+802.790536376"
Nov 03 19:44:49.536079 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:49.536026    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podStartSLOduration=1.536015954 podStartE2EDuration="1.536015954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:49.533537323 +0000 UTC m=+802.803660618" watchObservedRunningTime="2026-11-03 19:44:49.536015954 +0000 UTC m=+802.806139249"
Nov 03 19:44:49.731890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:49.731856    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:49.845252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:49.844317    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:44:49.845252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:49.844383    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:44:50.256198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:50.253702    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:50.256198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:50.253872    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:44:50.256198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:50.253919    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:44:50.398196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:50.397315    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:50.398196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:50.397454    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:50.398196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:50.397497    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 19:44:50.437244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:50.436723    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:50.726130 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:44:50.725912    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:44:50.831622 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:50.831541    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:44:50.831622 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:50.831606    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:44:51.129253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:51.128211    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:51.249584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:51.249546    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:44:51.249821 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:51.249794    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:44:51.393650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:51.393551    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:51.393907 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:51.393879    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 19:44:51.740139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:51.740045    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:51.834196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:51.833538    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:44:51.834196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:51.833612    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:44:52.249198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:52.248529    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:44:52.249198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:52.248592    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:44:52.393196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:52.392585    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:52.393196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:52.392652    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 19:44:52.447195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:52.443700    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:52.832134 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:52.832100    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:44:52.832561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:52.832149    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:44:53.066832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:53.066735    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:53.250204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:53.248830    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:44:53.250204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:53.248895    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:44:53.393196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:53.392582    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:53.393196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:53.392648    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 19:44:53.589254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:53.588304    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:53.630334 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:53.630308    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:53.735898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:53.735862    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:44:53.831633 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:53.831594    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:44:53.831827 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:53.831655    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:44:53.955196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:53.951867    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:54.258258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:54.246607    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:44:54.258258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:54.246683    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:44:54.393250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:54.392545    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:54.393250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:54.392620    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 19:44:54.722191 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:54.721532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:54.845200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:54.840799    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:44:54.845200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:54.840888    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 19:44:55.247062 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.247023    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:44:55.247460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.247096    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 19:44:55.351846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.351790    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:55.394952 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.394917    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:55.395237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.395202    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 19:44:55.837755 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.837722    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:55.838134 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.838111    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:55.838772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.838750    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 19:44:55.970910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.970874    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"]
Nov 03 19:44:55.978700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.976723    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:55.982554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.982462    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"]
Nov 03 19:44:55.982808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:55.982750    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:44:56.038101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.038044    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:56.121282 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.082387    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6158f102-r6mqq\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:56.121931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.121770    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:44:56.122220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.122193    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjqdc\" (UniqueName: \"kubernetes.io/projected/009fab84-7b8a-4199-9c69-b99aa4e41eeb-kube-api-access-mjqdc\") pod \"maas-api-revoke-sub-6158f102-r6mqq\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:56.122438 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.122415    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6158f102-r6mqq\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:56.223878 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.223833    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjqdc\" (UniqueName: \"kubernetes.io/projected/009fab84-7b8a-4199-9c69-b99aa4e41eeb-kube-api-access-mjqdc\") pod \"maas-api-revoke-sub-6158f102-r6mqq\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:56.224276 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.223919    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6158f102-r6mqq\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:56.224276 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.224119    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6158f102-r6mqq\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:56.225198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.225139    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6158f102-r6mqq\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:56.228664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.228638    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6158f102-r6mqq\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:56.232054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.232027    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjqdc\" (UniqueName: \"kubernetes.io/projected/009fab84-7b8a-4199-9c69-b99aa4e41eeb-kube-api-access-mjqdc\") pod \"maas-api-revoke-sub-6158f102-r6mqq\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:56.307973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.307934    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:56.344985 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.344959    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:56.345642 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.345602    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:56.346309 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.346286    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:44:56.503811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.502853    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:56.504480 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.504401    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:56.507467 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.507445    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 19:44:56.613646 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.613619    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"]
Nov 03 19:44:56.621007 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:44:56.620979    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009fab84_7b8a_4199_9c69_b99aa4e41eeb.slice/crio-84705eed0f54263107928812153fa4d9c4c0b46dce9a6ff18ceb5425874019e0 WatchSource:0}: Error finding container 84705eed0f54263107928812153fa4d9c4c0b46dce9a6ff18ceb5425874019e0: Status 404 returned error can't find the container with id 84705eed0f54263107928812153fa4d9c4c0b46dce9a6ff18ceb5425874019e0
Nov 03 19:44:56.742257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.742222    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:56.749389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:56.749354    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq" event={"ID":"009fab84-7b8a-4199-9c69-b99aa4e41eeb","Type":"ContainerStarted","Data":"84705eed0f54263107928812153fa4d9c4c0b46dce9a6ff18ceb5425874019e0"}
Nov 03 19:44:57.419607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:57.419552    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:57.760660 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:57.760574    4531 generic.go:356] "Generic (PLEG): container finished" podID="009fab84-7b8a-4199-9c69-b99aa4e41eeb" containerID="a6790e969cae82453acbc32397e466a887d714df9d0775458b47192c26971205" exitCode=0
Nov 03 19:44:57.762784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:57.762754    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq" event={"ID":"009fab84-7b8a-4199-9c69-b99aa4e41eeb","Type":"ContainerDied","Data":"a6790e969cae82453acbc32397e466a887d714df9d0775458b47192c26971205"}
Nov 03 19:44:57.858310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:57.858272    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:58.239870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:58.239836    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:58.919681 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:58.919656    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:44:58.955816 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:58.952948    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:59.085926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.085850    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-xks-ca\") pod \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") "
Nov 03 19:44:59.086065 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.085932    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-openshift-ca\") pod \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") "
Nov 03 19:44:59.086065 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.086034    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009fab84-7b8a-4199-9c69-b99aa4e41eeb-kube-api-access-mjqdc\" (UniqueName: \"kubernetes.io/projected/009fab84-7b8a-4199-9c69-b99aa4e41eeb-kube-api-access-mjqdc\") pod \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\" (UID: \"009fab84-7b8a-4199-9c69-b99aa4e41eeb\") "
Nov 03 19:44:59.086293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.086271    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-openshift-ca" pod "009fab84-7b8a-4199-9c69-b99aa4e41eeb" (UID: "009fab84-7b8a-4199-9c69-b99aa4e41eeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:59.086734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.086713    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:59.088002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.087982    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-xks-ca" pod "009fab84-7b8a-4199-9c69-b99aa4e41eeb" (UID: "009fab84-7b8a-4199-9c69-b99aa4e41eeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:59.088083 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.088059    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009fab84-7b8a-4199-9c69-b99aa4e41eeb-kube-api-access-mjqdc" pod "009fab84-7b8a-4199-9c69-b99aa4e41eeb" (UID: "009fab84-7b8a-4199-9c69-b99aa4e41eeb"). InnerVolumeSpecName "kube-api-access-mjqdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:59.188027 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.187995    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009fab84-7b8a-4199-9c69-b99aa4e41eeb-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:59.188027 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.188023    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjqdc\" (UniqueName: \"kubernetes.io/projected/009fab84-7b8a-4199-9c69-b99aa4e41eeb-kube-api-access-mjqdc\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:59.728522 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.728483    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:59.792196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.791921    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq" event={"ID":"009fab84-7b8a-4199-9c69-b99aa4e41eeb","Type":"ContainerDied","Data":"84705eed0f54263107928812153fa4d9c4c0b46dce9a6ff18ceb5425874019e0"}
Nov 03 19:44:59.792196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.792008    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84705eed0f54263107928812153fa4d9c4c0b46dce9a6ff18ceb5425874019e0"
Nov 03 19:44:59.792196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:44:59.792099    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"
Nov 03 19:45:00.139709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.139136    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"]
Nov 03 19:45:00.140118 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.140053    4531 cpu_manager.go:414] "containerMap: removing container" podUID="009fab84-7b8a-4199-9c69-b99aa4e41eeb" containerName="revoke-keys"
Nov 03 19:45:00.140118 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.140077    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009fab84-7b8a-4199-9c69-b99aa4e41eeb" containerName="revoke-keys"
Nov 03 19:45:00.140242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.140228    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="009fab84-7b8a-4199-9c69-b99aa4e41eeb" containerName="revoke-keys"
Nov 03 19:45:00.143710 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.143689    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"]
Nov 03 19:45:00.143833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.143813    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"
Nov 03 19:45:00.148325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.148284    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"]
Nov 03 19:45:00.148682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.148659    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"
Nov 03 19:45:00.152132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.152110    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:45:00.157141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.155684    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"]
Nov 03 19:45:00.157141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.155836    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"
Nov 03 19:45:00.159509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.158858    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"]
Nov 03 19:45:00.165465 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.165443    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"]
Nov 03 19:45:00.172779 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.172760    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"]
Nov 03 19:45:00.180226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.178404    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"
Nov 03 19:45:00.183010 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.182985    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"]
Nov 03 19:45:00.207813 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.207780    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffp52\" (UniqueName: \"kubernetes.io/projected/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-kube-api-access-ffp52\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9\" (UID: \"f7b2f030-ec61-42c3-b14d-371bbfef3b7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"
Nov 03 19:45:00.208015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.207930    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759pf\" (UniqueName: \"kubernetes.io/projected/cc0a1881-ed0c-488a-b9ef-e029167f5d48-kube-api-access-759pf\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp\" (UID: \"cc0a1881-ed0c-488a-b9ef-e029167f5d48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"
Nov 03 19:45:00.208015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.207963    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9\" (UID: \"f7b2f030-ec61-42c3-b14d-371bbfef3b7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"
Nov 03 19:45:00.208112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.208062    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqfwp\" (UniqueName: \"kubernetes.io/projected/84a59d90-5a71-43f2-a18a-49d29f0123b1-kube-api-access-gqfwp\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch\" (UID: \"84a59d90-5a71-43f2-a18a-49d29f0123b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"
Nov 03 19:45:00.208155 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.208137    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5799\" (UniqueName: \"kubernetes.io/projected/a398e3b7-2e39-4d05-baa0-1cf429addfc9-kube-api-access-h5799\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb\" (UID: \"a398e3b7-2e39-4d05-baa0-1cf429addfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"
Nov 03 19:45:00.208228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.208186    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0a1881-ed0c-488a-b9ef-e029167f5d48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp\" (UID: \"cc0a1881-ed0c-488a-b9ef-e029167f5d48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"
Nov 03 19:45:00.208344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.208325    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a59d90-5a71-43f2-a18a-49d29f0123b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch\" (UID: \"84a59d90-5a71-43f2-a18a-49d29f0123b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"
Nov 03 19:45:00.208400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.208354    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a398e3b7-2e39-4d05-baa0-1cf429addfc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb\" (UID: \"a398e3b7-2e39-4d05-baa0-1cf429addfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"
Nov 03 19:45:00.309492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.309459    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5799\" (UniqueName: \"kubernetes.io/projected/a398e3b7-2e39-4d05-baa0-1cf429addfc9-kube-api-access-h5799\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb\" (UID: \"a398e3b7-2e39-4d05-baa0-1cf429addfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"
Nov 03 19:45:00.309492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.309497    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0a1881-ed0c-488a-b9ef-e029167f5d48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp\" (UID: \"cc0a1881-ed0c-488a-b9ef-e029167f5d48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"
Nov 03 19:45:00.309682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.309565    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a59d90-5a71-43f2-a18a-49d29f0123b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch\" (UID: \"84a59d90-5a71-43f2-a18a-49d29f0123b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"
Nov 03 19:45:00.309682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.309581    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a398e3b7-2e39-4d05-baa0-1cf429addfc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb\" (UID: \"a398e3b7-2e39-4d05-baa0-1cf429addfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"
Nov 03 19:45:00.309682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.309665    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffp52\" (UniqueName: \"kubernetes.io/projected/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-kube-api-access-ffp52\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9\" (UID: \"f7b2f030-ec61-42c3-b14d-371bbfef3b7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"
Nov 03 19:45:00.309841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.309773    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759pf\" (UniqueName: \"kubernetes.io/projected/cc0a1881-ed0c-488a-b9ef-e029167f5d48-kube-api-access-759pf\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp\" (UID: \"cc0a1881-ed0c-488a-b9ef-e029167f5d48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"
Nov 03 19:45:00.309841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.309799    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9\" (UID: \"f7b2f030-ec61-42c3-b14d-371bbfef3b7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"
Nov 03 19:45:00.309947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.309856    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqfwp\" (UniqueName: \"kubernetes.io/projected/84a59d90-5a71-43f2-a18a-49d29f0123b1-kube-api-access-gqfwp\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch\" (UID: \"84a59d90-5a71-43f2-a18a-49d29f0123b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"
Nov 03 19:45:00.310323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.310302    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0a1881-ed0c-488a-b9ef-e029167f5d48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp\" (UID: \"cc0a1881-ed0c-488a-b9ef-e029167f5d48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"
Nov 03 19:45:00.310437 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.310420    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a398e3b7-2e39-4d05-baa0-1cf429addfc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb\" (UID: \"a398e3b7-2e39-4d05-baa0-1cf429addfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"
Nov 03 19:45:00.310473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.310445    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a59d90-5a71-43f2-a18a-49d29f0123b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch\" (UID: \"84a59d90-5a71-43f2-a18a-49d29f0123b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"
Nov 03 19:45:00.310648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.310631    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9\" (UID: \"f7b2f030-ec61-42c3-b14d-371bbfef3b7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"
Nov 03 19:45:00.317278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.317253    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffp52\" (UniqueName: \"kubernetes.io/projected/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-kube-api-access-ffp52\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9\" (UID: \"f7b2f030-ec61-42c3-b14d-371bbfef3b7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"
Nov 03 19:45:00.317399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.317348    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5799\" (UniqueName: \"kubernetes.io/projected/a398e3b7-2e39-4d05-baa0-1cf429addfc9-kube-api-access-h5799\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb\" (UID: \"a398e3b7-2e39-4d05-baa0-1cf429addfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"
Nov 03 19:45:00.317779 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.317759    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-759pf\" (UniqueName: \"kubernetes.io/projected/cc0a1881-ed0c-488a-b9ef-e029167f5d48-kube-api-access-759pf\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp\" (UID: \"cc0a1881-ed0c-488a-b9ef-e029167f5d48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"
Nov 03 19:45:00.317927 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.317907    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqfwp\" (UniqueName: \"kubernetes.io/projected/84a59d90-5a71-43f2-a18a-49d29f0123b1-kube-api-access-gqfwp\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch\" (UID: \"84a59d90-5a71-43f2-a18a-49d29f0123b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"
Nov 03 19:45:00.481429 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.481337    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"
Nov 03 19:45:00.484510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.484483    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"
Nov 03 19:45:00.493274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.493250    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"
Nov 03 19:45:00.507982 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.507918    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"
Nov 03 19:45:00.542371 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.532230    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:00.818312 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.818282    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"]
Nov 03 19:45:00.818618 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:00.818592    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda398e3b7_2e39_4d05_baa0_1cf429addfc9.slice/crio-4bef1555ccddc4db8c12516459ea9ba30d6d3f9c8660150d669f5ea97d004a5b WatchSource:0}: Error finding container 4bef1555ccddc4db8c12516459ea9ba30d6d3f9c8660150d669f5ea97d004a5b: Status 404 returned error can't find the container with id 4bef1555ccddc4db8c12516459ea9ba30d6d3f9c8660150d669f5ea97d004a5b
Nov 03 19:45:00.868465 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.868386    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"]
Nov 03 19:45:00.868465 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:00.868454    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0a1881_ed0c_488a_b9ef_e029167f5d48.slice/crio-795c29e11526e18c508027e32adbf37e5bc839563c13c9653cc14e4cd020a055 WatchSource:0}: Error finding container 795c29e11526e18c508027e32adbf37e5bc839563c13c9653cc14e4cd020a055: Status 404 returned error can't find the container with id 795c29e11526e18c508027e32adbf37e5bc839563c13c9653cc14e4cd020a055
Nov 03 19:45:00.871905 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:00.871865    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b2f030_ec61_42c3_b14d_371bbfef3b7a.slice/crio-f6f2b2cbad19a8843b2205680b50a30d2e61e9045161ee916b89a416e5c58d20 WatchSource:0}: Error finding container f6f2b2cbad19a8843b2205680b50a30d2e61e9045161ee916b89a416e5c58d20: Status 404 returned error can't find the container with id f6f2b2cbad19a8843b2205680b50a30d2e61e9045161ee916b89a416e5c58d20
Nov 03 19:45:00.873909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.873886    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"]
Nov 03 19:45:00.890622 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:00.890598    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"]
Nov 03 19:45:00.893032 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:00.892995    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a59d90_5a71_43f2_a18a_49d29f0123b1.slice/crio-c6d7745a698ada3a5b769c53d2ed9cd29213890906c2f783a1e959896da2d25a WatchSource:0}: Error finding container c6d7745a698ada3a5b769c53d2ed9cd29213890906c2f783a1e959896da2d25a: Status 404 returned error can't find the container with id c6d7745a698ada3a5b769c53d2ed9cd29213890906c2f783a1e959896da2d25a
Nov 03 19:45:01.196537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.196504    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"]
Nov 03 19:45:01.202906 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.202878    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:01.206067 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.206039    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:45:01.207276 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.207247    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"]
Nov 03 19:45:01.330934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.330904    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:01.333408 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.333387    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfpzj\" (UniqueName: \"kubernetes.io/projected/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-kube-api-access-zfpzj\") pod \"maas-api-revoke-sub-c764a998-c4rfm\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:01.333519 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.333461    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c764a998-c4rfm\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:01.333557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.333536    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c764a998-c4rfm\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:01.434720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.434686    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c764a998-c4rfm\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:01.434720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.434725    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c764a998-c4rfm\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:01.435000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.434899    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfpzj\" (UniqueName: \"kubernetes.io/projected/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-kube-api-access-zfpzj\") pod \"maas-api-revoke-sub-c764a998-c4rfm\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:01.435374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.435352    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c764a998-c4rfm\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:01.437453 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.437430    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c764a998-c4rfm\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:01.448367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.448306    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfpzj\" (UniqueName: \"kubernetes.io/projected/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-kube-api-access-zfpzj\") pod \"maas-api-revoke-sub-c764a998-c4rfm\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:01.521379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.521351    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:01.803367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.803340    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"]
Nov 03 19:45:01.808149 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.808119    4531 generic.go:356] "Generic (PLEG): container finished" podID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerID="7f2fb81a2e405b143abb671d19847cefc3e4a333f439df1d70bbc4ee760cb59b" exitCode=60
Nov 03 19:45:01.808303 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.808223    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch" event={"ID":"84a59d90-5a71-43f2-a18a-49d29f0123b1","Type":"ContainerDied","Data":"7f2fb81a2e405b143abb671d19847cefc3e4a333f439df1d70bbc4ee760cb59b"}
Nov 03 19:45:01.808360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.808300    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch" event={"ID":"84a59d90-5a71-43f2-a18a-49d29f0123b1","Type":"ContainerStarted","Data":"c6d7745a698ada3a5b769c53d2ed9cd29213890906c2f783a1e959896da2d25a"}
Nov 03 19:45:01.808360 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:01.808309    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc192a79_2ff6_43cb_a4a4_9ecf03c76d02.slice/crio-165973c3562cfa4dbba247def61c16746b6bef23ecab7b44fb23145e53985307 WatchSource:0}: Error finding container 165973c3562cfa4dbba247def61c16746b6bef23ecab7b44fb23145e53985307: Status 404 returned error can't find the container with id 165973c3562cfa4dbba247def61c16746b6bef23ecab7b44fb23145e53985307
Nov 03 19:45:01.808671 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.808655    4531 scope.go:122] "RemoveContainer" containerID="7f2fb81a2e405b143abb671d19847cefc3e4a333f439df1d70bbc4ee760cb59b"
Nov 03 19:45:01.810443 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.810219    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" event={"ID":"a398e3b7-2e39-4d05-baa0-1cf429addfc9","Type":"ContainerStarted","Data":"15350995cd2108f2fa2a49ac567f965762d2a32c2687f90f805115b5d7e5875f"}
Nov 03 19:45:01.810443 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.810270    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" event={"ID":"a398e3b7-2e39-4d05-baa0-1cf429addfc9","Type":"ContainerStarted","Data":"4bef1555ccddc4db8c12516459ea9ba30d6d3f9c8660150d669f5ea97d004a5b"}
Nov 03 19:45:01.812941 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.812919    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" event={"ID":"f7b2f030-ec61-42c3-b14d-371bbfef3b7a","Type":"ContainerStarted","Data":"8473c7aa259510d52ff819a4933b753d1685f35c16d9c4e5f9222b07db9f78d4"}
Nov 03 19:45:01.813030 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.812966    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" event={"ID":"f7b2f030-ec61-42c3-b14d-371bbfef3b7a","Type":"ContainerStarted","Data":"f6f2b2cbad19a8843b2205680b50a30d2e61e9045161ee916b89a416e5c58d20"}
Nov 03 19:45:01.814564 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.814543    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" event={"ID":"cc0a1881-ed0c-488a-b9ef-e029167f5d48","Type":"ContainerStarted","Data":"ec54fdb16cd1ac715c15bb22bb9015dcc08adbee3d8ffed46e1b467e194cfcb3"}
Nov 03 19:45:01.814654 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.814579    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" event={"ID":"cc0a1881-ed0c-488a-b9ef-e029167f5d48","Type":"ContainerStarted","Data":"795c29e11526e18c508027e32adbf37e5bc839563c13c9653cc14e4cd020a055"}
Nov 03 19:45:01.850185 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.849772    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" podStartSLOduration=1.849747032 podStartE2EDuration="1.849747032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:01.836738321 +0000 UTC m=+815.106861638" watchObservedRunningTime="2026-11-03 19:45:01.849747032 +0000 UTC m=+815.119870328"
Nov 03 19:45:01.856219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.856147    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" podStartSLOduration=1.8561301590000001 podStartE2EDuration="1.856130159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:01.849962446 +0000 UTC m=+815.120085741" watchObservedRunningTime="2026-11-03 19:45:01.856130159 +0000 UTC m=+815.126253454"
Nov 03 19:45:01.866422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:01.866380    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" podStartSLOduration=1.866366847 podStartE2EDuration="1.866366847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:01.862140025 +0000 UTC m=+815.132263322" watchObservedRunningTime="2026-11-03 19:45:01.866366847 +0000 UTC m=+815.136490165"
Nov 03 19:45:02.192605 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.186741    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:02.527879 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.527824    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"]
Nov 03 19:45:02.534041 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.534014    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.555374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.555343    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"]
Nov 03 19:45:02.574366 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.574326    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.574503 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.574385    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.574503 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.574441    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.574503 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.574481    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.574637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.574504    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.574637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.574560    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-565dx\" (UniqueName: \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-kube-api-access-565dx\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.574737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.574720    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.574789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.574756    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.574833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.574800    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.675821 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.675778    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.676112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.675835    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.676112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.675877    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.676112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.675915    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.676112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.675949    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.676505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.676132    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.676505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.676212    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.676505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.676239    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.676505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.676292    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.676505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.676313    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-565dx\" (UniqueName: \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-kube-api-access-565dx\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.676505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.676387    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.677022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.676596    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.677022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.676706    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.677022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.676946    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.678814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.678788    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.678898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.678847    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.683360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.683338    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-565dx\" (UniqueName: \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-kube-api-access-565dx\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.683930 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.683914    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.826133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.826101    4531 generic.go:356] "Generic (PLEG): container finished" podID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerID="2b10966da9dcb42a9dabedda531035ddadb4c40d27331e0870646fa680b47c8f" exitCode=60
Nov 03 19:45:02.826270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.826211    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch" event={"ID":"84a59d90-5a71-43f2-a18a-49d29f0123b1","Type":"ContainerDied","Data":"2b10966da9dcb42a9dabedda531035ddadb4c40d27331e0870646fa680b47c8f"}
Nov 03 19:45:02.826336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.826275    4531 scope.go:122] "RemoveContainer" containerID="7f2fb81a2e405b143abb671d19847cefc3e4a333f439df1d70bbc4ee760cb59b"
Nov 03 19:45:02.826655 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.826639    4531 scope.go:122] "RemoveContainer" containerID="2b10966da9dcb42a9dabedda531035ddadb4c40d27331e0870646fa680b47c8f"
Nov 03 19:45:02.826983 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:02.826950    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch_odh-ai-gateway-infra(84a59d90-5a71-43f2-a18a-49d29f0123b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1"
Nov 03 19:45:02.828133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.828107    4531 generic.go:356] "Generic (PLEG): container finished" podID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerID="15350995cd2108f2fa2a49ac567f965762d2a32c2687f90f805115b5d7e5875f" exitCode=60
Nov 03 19:45:02.828250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.828201    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" event={"ID":"a398e3b7-2e39-4d05-baa0-1cf429addfc9","Type":"ContainerDied","Data":"15350995cd2108f2fa2a49ac567f965762d2a32c2687f90f805115b5d7e5875f"}
Nov 03 19:45:02.828492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.828474    4531 scope.go:122] "RemoveContainer" containerID="15350995cd2108f2fa2a49ac567f965762d2a32c2687f90f805115b5d7e5875f"
Nov 03 19:45:02.830801 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.830771    4531 generic.go:356] "Generic (PLEG): container finished" podID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerID="8473c7aa259510d52ff819a4933b753d1685f35c16d9c4e5f9222b07db9f78d4" exitCode=60
Nov 03 19:45:02.830910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.830801    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" event={"ID":"f7b2f030-ec61-42c3-b14d-371bbfef3b7a","Type":"ContainerDied","Data":"8473c7aa259510d52ff819a4933b753d1685f35c16d9c4e5f9222b07db9f78d4"}
Nov 03 19:45:02.831092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.831075    4531 scope.go:122] "RemoveContainer" containerID="8473c7aa259510d52ff819a4933b753d1685f35c16d9c4e5f9222b07db9f78d4"
Nov 03 19:45:02.833764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.833743    4531 generic.go:356] "Generic (PLEG): container finished" podID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerID="ec54fdb16cd1ac715c15bb22bb9015dcc08adbee3d8ffed46e1b467e194cfcb3" exitCode=60
Nov 03 19:45:02.833938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.833890    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" event={"ID":"cc0a1881-ed0c-488a-b9ef-e029167f5d48","Type":"ContainerDied","Data":"ec54fdb16cd1ac715c15bb22bb9015dcc08adbee3d8ffed46e1b467e194cfcb3"}
Nov 03 19:45:02.834161 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.834075    4531 scope.go:122] "RemoveContainer" containerID="ec54fdb16cd1ac715c15bb22bb9015dcc08adbee3d8ffed46e1b467e194cfcb3"
Nov 03 19:45:02.835873 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.835815    4531 generic.go:356] "Generic (PLEG): container finished" podID="bc192a79-2ff6-43cb-a4a4-9ecf03c76d02" containerID="fd64d7710557c4cbbf7bbd647f0e0b5341e056ae5cd376d334112857d74e718a" exitCode=0
Nov 03 19:45:02.835911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.835890    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm" event={"ID":"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02","Type":"ContainerDied","Data":"fd64d7710557c4cbbf7bbd647f0e0b5341e056ae5cd376d334112857d74e718a"}
Nov 03 19:45:02.835944 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.835926    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm" event={"ID":"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02","Type":"ContainerStarted","Data":"165973c3562cfa4dbba247def61c16746b6bef23ecab7b44fb23145e53985307"}
Nov 03 19:45:02.855340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.855312    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:02.916738 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.916341    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"]
Nov 03 19:45:02.926507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.925822    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:02.949499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.949407    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"]
Nov 03 19:45:02.967877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.966073    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:02.979804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.979776    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7wbr\" (UniqueName: \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-kube-api-access-r7wbr\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:02.980036 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.980001    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:02.980140 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.980118    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5789cd8c-de84-4996-b0bf-6c72446833db-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:02.980624 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.980602    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:02.981288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.980837    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:02.981288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.980907    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5789cd8c-de84-4996-b0bf-6c72446833db-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:02.981288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.981081    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:02.981288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.981216    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:02.981529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:02.981308    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.130950    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.159955    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5789cd8c-de84-4996-b0bf-6c72446833db-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.160011    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.160071    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.160095    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.160281    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7wbr\" (UniqueName: \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-kube-api-access-r7wbr\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.160364    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.160416    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5789cd8c-de84-4996-b0bf-6c72446833db-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.160439    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.161050    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.161102    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.163807    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5789cd8c-de84-4996-b0bf-6c72446833db-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.164861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.164377    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.168849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.168817    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.183847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.182965    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5789cd8c-de84-4996-b0bf-6c72446833db-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.183847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.183222    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.189580 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.189543    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7wbr\" (UniqueName: \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-kube-api-access-r7wbr\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.190791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.190767    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.203192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.203152    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:03.205108 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.204191    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"]
Nov 03 19:45:03.208728 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.208699    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.208808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.208758    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.208808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.208788    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.271887 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.271867    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:03.574401 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:03.574365    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5789cd8c_de84_4996_b0bf_6c72446833db.slice/crio-d4729fc47a7d7681eb6e854511da95d3709c221c63ae3ff5ea97035048335f1a WatchSource:0}: Error finding container d4729fc47a7d7681eb6e854511da95d3709c221c63ae3ff5ea97035048335f1a: Status 404 returned error can't find the container with id d4729fc47a7d7681eb6e854511da95d3709c221c63ae3ff5ea97035048335f1a
Nov 03 19:45:03.574843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.574476    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"]
Nov 03 19:45:03.577461 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.577422    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.577584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.577532    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.577584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.577571    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.936582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:03.936532    4531 scope.go:122] "RemoveContainer" containerID="2b10966da9dcb42a9dabedda531035ddadb4c40d27331e0870646fa680b47c8f"
Nov 03 19:45:03.937289 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:03.937254    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch_odh-ai-gateway-infra(84a59d90-5a71-43f2-a18a-49d29f0123b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1"
Nov 03 19:45:04.013993 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.013954    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:04.071584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.071536    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" event={"ID":"a398e3b7-2e39-4d05-baa0-1cf429addfc9","Type":"ContainerStarted","Data":"e4867c29bd2c42e70dab3aa1d45c677c417152f39c2816deb68b344f9a0690c7"}
Nov 03 19:45:04.072509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.072476    4531 scope.go:122] "RemoveContainer" containerID="e4867c29bd2c42e70dab3aa1d45c677c417152f39c2816deb68b344f9a0690c7"
Nov 03 19:45:04.081250 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:04.078556    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb_odh-ai-gateway-infra(a398e3b7-2e39-4d05-baa0-1cf429addfc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9"
Nov 03 19:45:04.214193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.193016    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" event={"ID":"5789cd8c-de84-4996-b0bf-6c72446833db","Type":"ContainerStarted","Data":"770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b"}
Nov 03 19:45:04.214193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.193082    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" event={"ID":"5789cd8c-de84-4996-b0bf-6c72446833db","Type":"ContainerStarted","Data":"d4729fc47a7d7681eb6e854511da95d3709c221c63ae3ff5ea97035048335f1a"}
Nov 03 19:45:04.235094 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.231062    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" podStartSLOduration=2.2310422819999998 podStartE2EDuration="2.231042282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:04.229848023 +0000 UTC m=+817.499971331" watchObservedRunningTime="2026-11-03 19:45:04.231042282 +0000 UTC m=+817.501165577"
Nov 03 19:45:04.253151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.253112    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" event={"ID":"f7b2f030-ec61-42c3-b14d-371bbfef3b7a","Type":"ContainerStarted","Data":"12a661f89b596aa344c9a6691a78375a628ce3fe01b533471caa2596be18f30a"}
Nov 03 19:45:04.270197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.253865    4531 scope.go:122] "RemoveContainer" containerID="12a661f89b596aa344c9a6691a78375a628ce3fe01b533471caa2596be18f30a"
Nov 03 19:45:04.272341 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:04.272303    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9_odh-ai-gateway-infra(f7b2f030-ec61-42c3-b14d-371bbfef3b7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a"
Nov 03 19:45:04.289371 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.281638    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:04.289741 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.289653    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:45:04.289896 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.289874    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:45:04.291138 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.291105    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" event={"ID":"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7","Type":"ContainerStarted","Data":"daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464"}
Nov 03 19:45:04.291365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.291337    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" event={"ID":"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7","Type":"ContainerStarted","Data":"e2bdaed4451b45de2369476555dfc08a7ca50ec1372203671d40cb863e3d151a"}
Nov 03 19:45:04.323120 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.312550    4531 scope.go:122] "RemoveContainer" containerID="36e41f1d959b4a89461bb020581ce99348b871ca95969cf7cff4adfdf102c0dc"
Nov 03 19:45:04.323120 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:04.322833    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp_odh-ai-gateway-infra(cc0a1881-ed0c-488a-b9ef-e029167f5d48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48"
Nov 03 19:45:04.323120 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.315218    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" event={"ID":"cc0a1881-ed0c-488a-b9ef-e029167f5d48","Type":"ContainerStarted","Data":"36e41f1d959b4a89461bb020581ce99348b871ca95969cf7cff4adfdf102c0dc"}
Nov 03 19:45:04.326040 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.325486    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" podStartSLOduration=2.3254687 podStartE2EDuration="2.3254687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:04.320956718 +0000 UTC m=+817.591080005" watchObservedRunningTime="2026-11-03 19:45:04.3254687 +0000 UTC m=+817.595591996"
Nov 03 19:45:04.828474 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.827421    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:04.905729 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:04.858590    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:05.173000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.116427    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 19:45:05.173000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.116509    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 19:45:05.180035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.179624    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:05.333343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.328638    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-xks-ca\") pod \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") "
Nov 03 19:45:05.397591 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.350006    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-kube-api-access-zfpzj\" (UniqueName: \"kubernetes.io/projected/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-kube-api-access-zfpzj\") pod \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") "
Nov 03 19:45:05.397591 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.336377    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:45:05.397591 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.350240    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-openshift-ca\") pod \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\" (UID: \"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02\") "
Nov 03 19:45:05.397591 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.350350    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:45:05.397591 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.361658    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-openshift-ca" pod "bc192a79-2ff6-43cb-a4a4-9ecf03c76d02" (UID: "bc192a79-2ff6-43cb-a4a4-9ecf03c76d02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:05.505810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.505468    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-xks-ca" pod "bc192a79-2ff6-43cb-a4a4-9ecf03c76d02" (UID: "bc192a79-2ff6-43cb-a4a4-9ecf03c76d02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:05.513010 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.512042    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:05.559197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.544568    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-kube-api-access-zfpzj" pod "bc192a79-2ff6-43cb-a4a4-9ecf03c76d02" (UID: "bc192a79-2ff6-43cb-a4a4-9ecf03c76d02"). InnerVolumeSpecName "kube-api-access-zfpzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:05.559197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.554222    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:05.599977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.599933    4531 generic.go:356] "Generic (PLEG): container finished" podID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerID="e4867c29bd2c42e70dab3aa1d45c677c417152f39c2816deb68b344f9a0690c7" exitCode=60
Nov 03 19:45:05.600415 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.600385    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" event={"ID":"a398e3b7-2e39-4d05-baa0-1cf429addfc9","Type":"ContainerDied","Data":"e4867c29bd2c42e70dab3aa1d45c677c417152f39c2816deb68b344f9a0690c7"}
Nov 03 19:45:05.600606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.600587    4531 scope.go:122] "RemoveContainer" containerID="15350995cd2108f2fa2a49ac567f965762d2a32c2687f90f805115b5d7e5875f"
Nov 03 19:45:05.601300 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.601280    4531 scope.go:122] "RemoveContainer" containerID="e4867c29bd2c42e70dab3aa1d45c677c417152f39c2816deb68b344f9a0690c7"
Nov 03 19:45:05.601857 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:05.601829    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb_odh-ai-gateway-infra(a398e3b7-2e39-4d05-baa0-1cf429addfc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9"
Nov 03 19:45:05.626639 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.622155    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:45:05.657109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.655860    4531 generic.go:356] "Generic (PLEG): container finished" podID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerID="12a661f89b596aa344c9a6691a78375a628ce3fe01b533471caa2596be18f30a" exitCode=60
Nov 03 19:45:05.657109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.655989    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" event={"ID":"f7b2f030-ec61-42c3-b14d-371bbfef3b7a","Type":"ContainerDied","Data":"12a661f89b596aa344c9a6691a78375a628ce3fe01b533471caa2596be18f30a"}
Nov 03 19:45:05.657109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.656468    4531 scope.go:122] "RemoveContainer" containerID="12a661f89b596aa344c9a6691a78375a628ce3fe01b533471caa2596be18f30a"
Nov 03 19:45:05.657109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.656504    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfpzj\" (UniqueName: \"kubernetes.io/projected/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02-kube-api-access-zfpzj\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:05.657109 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:05.656783    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9_odh-ai-gateway-infra(f7b2f030-ec61-42c3-b14d-371bbfef3b7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a"
Nov 03 19:45:05.672206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.669350    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:05.709625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.701072    4531 generic.go:356] "Generic (PLEG): container finished" podID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerID="36e41f1d959b4a89461bb020581ce99348b871ca95969cf7cff4adfdf102c0dc" exitCode=60
Nov 03 19:45:05.709625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.701251    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" event={"ID":"cc0a1881-ed0c-488a-b9ef-e029167f5d48","Type":"ContainerDied","Data":"36e41f1d959b4a89461bb020581ce99348b871ca95969cf7cff4adfdf102c0dc"}
Nov 03 19:45:05.709625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.701748    4531 scope.go:122] "RemoveContainer" containerID="36e41f1d959b4a89461bb020581ce99348b871ca95969cf7cff4adfdf102c0dc"
Nov 03 19:45:05.709625 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:05.702157    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp_odh-ai-gateway-infra(cc0a1881-ed0c-488a-b9ef-e029167f5d48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48"
Nov 03 19:45:05.743191 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.743146    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"
Nov 03 19:45:05.750354 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.744950    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm" event={"ID":"bc192a79-2ff6-43cb-a4a4-9ecf03c76d02","Type":"ContainerDied","Data":"165973c3562cfa4dbba247def61c16746b6bef23ecab7b44fb23145e53985307"}
Nov 03 19:45:05.750354 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.745006    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="165973c3562cfa4dbba247def61c16746b6bef23ecab7b44fb23145e53985307"
Nov 03 19:45:05.750354 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.748252    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:05.817264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.812275    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:05.910347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.910312    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 19:45:05.910945 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.910914    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 19:45:05.956385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:05.956343    4531 scope.go:122] "RemoveContainer" containerID="8473c7aa259510d52ff819a4933b753d1685f35c16d9c4e5f9222b07db9f78d4"
Nov 03 19:45:06.013337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.013292    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:45:06.056788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.056740    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:06.171546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.171240    4531 scope.go:122] "RemoveContainer" containerID="ec54fdb16cd1ac715c15bb22bb9015dcc08adbee3d8ffed46e1b467e194cfcb3"
Nov 03 19:45:06.327224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.326757    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:06.327224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.327209    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:45:06.327448 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.327251    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:45:06.358199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.338290    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:06.382423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.382377    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:45:06.441600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.415248    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Nov 03 19:45:06.531223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.531167    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:06.659693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.659652    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:06.811384 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.811248    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:06.813415 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.813392    4531 scope.go:122] "RemoveContainer" containerID="e4867c29bd2c42e70dab3aa1d45c677c417152f39c2816deb68b344f9a0690c7"
Nov 03 19:45:06.813659 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:06.813637    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb_odh-ai-gateway-infra(a398e3b7-2e39-4d05-baa0-1cf429addfc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9"
Nov 03 19:45:06.816032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.816003    4531 scope.go:122] "RemoveContainer" containerID="12a661f89b596aa344c9a6691a78375a628ce3fe01b533471caa2596be18f30a"
Nov 03 19:45:06.816310 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:06.816285    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9_odh-ai-gateway-infra(f7b2f030-ec61-42c3-b14d-371bbfef3b7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a"
Nov 03 19:45:06.817774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.817756    4531 scope.go:122] "RemoveContainer" containerID="36e41f1d959b4a89461bb020581ce99348b871ca95969cf7cff4adfdf102c0dc"
Nov 03 19:45:06.817949 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:06.817930    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp_odh-ai-gateway-infra(cc0a1881-ed0c-488a-b9ef-e029167f5d48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48"
Nov 03 19:45:06.861723 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.861685    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 19:45:06.861994 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:06.861960    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 19:45:07.008361 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:07.008321    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:07.284083 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:07.284050    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:45:07.293237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:07.289288    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:45:07.867457 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:07.867424    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 19:45:07.867763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:07.867732    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 19:45:08.272621 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:08.272584    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:45:08.284224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:08.283268    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:45:08.868580 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:08.868542    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 19:45:08.868887 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:08.868854    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 19:45:09.144194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:09.131879    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:09.280199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:09.276547    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:45:09.280199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:09.276611    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:45:09.860198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:09.855980    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 03 19:45:09.860198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:09.856036    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 03 19:45:10.272728 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.272691    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:45:10.272923 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.272745    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:45:10.600233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.600140    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"]
Nov 03 19:45:10.600776 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.600763    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bc192a79-2ff6-43cb-a4a4-9ecf03c76d02" containerName="revoke-keys"
Nov 03 19:45:10.600824 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.600777    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc192a79-2ff6-43cb-a4a4-9ecf03c76d02" containerName="revoke-keys"
Nov 03 19:45:10.600924 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.600915    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc192a79-2ff6-43cb-a4a4-9ecf03c76d02" containerName="revoke-keys"
Nov 03 19:45:10.604631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.604603    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:10.607586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.607564    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:45:10.610664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.610641    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"]
Nov 03 19:45:10.697804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.697765    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pks\" (UniqueName: \"kubernetes.io/projected/716a697b-354d-4e08-baeb-b4a3aed59613-kube-api-access-h6pks\") pod \"maas-api-revoke-sub-2eef7ec0-5cqw8\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:10.698013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.697883    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eef7ec0-5cqw8\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:10.698013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.697960    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eef7ec0-5cqw8\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:10.799096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.799060    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pks\" (UniqueName: \"kubernetes.io/projected/716a697b-354d-4e08-baeb-b4a3aed59613-kube-api-access-h6pks\") pod \"maas-api-revoke-sub-2eef7ec0-5cqw8\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:10.799336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.799216    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eef7ec0-5cqw8\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:10.799336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.799283    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eef7ec0-5cqw8\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:10.800020 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.799992    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eef7ec0-5cqw8\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:10.801626 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.801598    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eef7ec0-5cqw8\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:10.806029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.806008    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pks\" (UniqueName: \"kubernetes.io/projected/716a697b-354d-4e08-baeb-b4a3aed59613-kube-api-access-h6pks\") pod \"maas-api-revoke-sub-2eef7ec0-5cqw8\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:10.924260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:10.924217    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:11.136955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:11.135749    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:11.232234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:11.232201    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"]
Nov 03 19:45:11.272951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:11.272920    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:45:11.273092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:11.272976    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 19:45:11.486017 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:11.485927    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:11.487106 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:11.486995    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:11.489196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:11.488447    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:45:11.902184 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:11.902136    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8" event={"ID":"716a697b-354d-4e08-baeb-b4a3aed59613","Type":"ContainerStarted","Data":"6d09075dd9c334a28de69858c5b21ddf597cd8bbf3265dd8df8b2a9542513360"}
Nov 03 19:45:11.902643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:11.902211    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8" event={"ID":"716a697b-354d-4e08-baeb-b4a3aed59613","Type":"ContainerStarted","Data":"e1aa4826a3f0821711b4d3a2c5c31a588cf55d986d0cf4a2ec308f66a6821891"}
Nov 03 19:45:11.917060 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:11.916392    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8" podStartSLOduration=1.916374293 podStartE2EDuration="1.916374293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:11.915471587 +0000 UTC m=+825.185594885" watchObservedRunningTime="2026-11-03 19:45:11.916374293 +0000 UTC m=+825.186497589"
Nov 03 19:45:12.277335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:12.277308    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:12.277633 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:12.277608    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:12.278360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:12.278341    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:45:12.843607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:12.842598    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:12.911307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:12.911276    4531 generic.go:356] "Generic (PLEG): container finished" podID="716a697b-354d-4e08-baeb-b4a3aed59613" containerID="6d09075dd9c334a28de69858c5b21ddf597cd8bbf3265dd8df8b2a9542513360" exitCode=0
Nov 03 19:45:12.911776 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:12.911366    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8" event={"ID":"716a697b-354d-4e08-baeb-b4a3aed59613","Type":"ContainerDied","Data":"6d09075dd9c334a28de69858c5b21ddf597cd8bbf3265dd8df8b2a9542513360"}
Nov 03 19:45:14.108790 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.108762    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:14.199307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.199218    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-openshift-ca\") pod \"716a697b-354d-4e08-baeb-b4a3aed59613\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") "
Nov 03 19:45:14.199466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.199366    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-xks-ca\") pod \"716a697b-354d-4e08-baeb-b4a3aed59613\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") "
Nov 03 19:45:14.199550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.199533    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716a697b-354d-4e08-baeb-b4a3aed59613-kube-api-access-h6pks\" (UniqueName: \"kubernetes.io/projected/716a697b-354d-4e08-baeb-b4a3aed59613-kube-api-access-h6pks\") pod \"716a697b-354d-4e08-baeb-b4a3aed59613\" (UID: \"716a697b-354d-4e08-baeb-b4a3aed59613\") "
Nov 03 19:45:14.199638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.199618    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-openshift-ca" pod "716a697b-354d-4e08-baeb-b4a3aed59613" (UID: "716a697b-354d-4e08-baeb-b4a3aed59613"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:14.200141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.200116    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:14.201575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.201546    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716a697b-354d-4e08-baeb-b4a3aed59613-kube-api-access-h6pks" pod "716a697b-354d-4e08-baeb-b4a3aed59613" (UID: "716a697b-354d-4e08-baeb-b4a3aed59613"). InnerVolumeSpecName "kube-api-access-h6pks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:14.201575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.201549    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-xks-ca" pod "716a697b-354d-4e08-baeb-b4a3aed59613" (UID: "716a697b-354d-4e08-baeb-b4a3aed59613"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:14.253084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.253031    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:14.302123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.302087    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716a697b-354d-4e08-baeb-b4a3aed59613-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:14.302123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.302122    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6pks\" (UniqueName: \"kubernetes.io/projected/716a697b-354d-4e08-baeb-b4a3aed59613-kube-api-access-h6pks\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:14.927204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.927148    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"
Nov 03 19:45:14.927355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.927212    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8" event={"ID":"716a697b-354d-4e08-baeb-b4a3aed59613","Type":"ContainerDied","Data":"e1aa4826a3f0821711b4d3a2c5c31a588cf55d986d0cf4a2ec308f66a6821891"}
Nov 03 19:45:14.927355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:14.927265    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1aa4826a3f0821711b4d3a2c5c31a588cf55d986d0cf4a2ec308f66a6821891"
Nov 03 19:45:15.293576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:15.293530    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:15.431020 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:15.430979    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:16.377058 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:16.377021    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:16.518204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:16.516310    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:16.856595 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:16.856568    4531 scope.go:122] "RemoveContainer" containerID="2b10966da9dcb42a9dabedda531035ddadb4c40d27331e0870646fa680b47c8f"
Nov 03 19:45:16.856884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:16.856862    4531 scope.go:122] "RemoveContainer" containerID="36e41f1d959b4a89461bb020581ce99348b871ca95969cf7cff4adfdf102c0dc"
Nov 03 19:45:16.856948 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:16.856905    4531 scope.go:122] "RemoveContainer" containerID="e4867c29bd2c42e70dab3aa1d45c677c417152f39c2816deb68b344f9a0690c7"
Nov 03 19:45:17.333805 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.333765    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:17.851533 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.851510    4531 scope.go:122] "RemoveContainer" containerID="12a661f89b596aa344c9a6691a78375a628ce3fe01b533471caa2596be18f30a"
Nov 03 19:45:17.951891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.951864    4531 generic.go:356] "Generic (PLEG): container finished" podID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerID="dc401d5505ace029f6009926388a489721314e541d3389404d0205187a814a64" exitCode=60
Nov 03 19:45:17.951979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.951933    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" event={"ID":"cc0a1881-ed0c-488a-b9ef-e029167f5d48","Type":"ContainerDied","Data":"dc401d5505ace029f6009926388a489721314e541d3389404d0205187a814a64"}
Nov 03 19:45:17.952022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.951996    4531 scope.go:122] "RemoveContainer" containerID="36e41f1d959b4a89461bb020581ce99348b871ca95969cf7cff4adfdf102c0dc"
Nov 03 19:45:17.952487 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.952465    4531 scope.go:122] "RemoveContainer" containerID="dc401d5505ace029f6009926388a489721314e541d3389404d0205187a814a64"
Nov 03 19:45:17.952817 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:17.952795    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp_odh-ai-gateway-infra(cc0a1881-ed0c-488a-b9ef-e029167f5d48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48"
Nov 03 19:45:17.954352 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.954329    4531 generic.go:356] "Generic (PLEG): container finished" podID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerID="3e91efdb41e4d036161e4a5f14b2ae556f7b543c783f45086aba6fae8d6426d2" exitCode=60
Nov 03 19:45:17.954443 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.954400    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch" event={"ID":"84a59d90-5a71-43f2-a18a-49d29f0123b1","Type":"ContainerDied","Data":"3e91efdb41e4d036161e4a5f14b2ae556f7b543c783f45086aba6fae8d6426d2"}
Nov 03 19:45:17.954772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.954754    4531 scope.go:122] "RemoveContainer" containerID="3e91efdb41e4d036161e4a5f14b2ae556f7b543c783f45086aba6fae8d6426d2"
Nov 03 19:45:17.955097 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:17.955076    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch_odh-ai-gateway-infra(84a59d90-5a71-43f2-a18a-49d29f0123b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1"
Nov 03 19:45:17.956342 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.956320    4531 generic.go:356] "Generic (PLEG): container finished" podID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerID="c280a24f54eb0d6fcdeb88bdc6511d8ab2691443cfc608ee3c6e46f3c9edfd60" exitCode=60
Nov 03 19:45:17.956439 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.956357    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" event={"ID":"a398e3b7-2e39-4d05-baa0-1cf429addfc9","Type":"ContainerDied","Data":"c280a24f54eb0d6fcdeb88bdc6511d8ab2691443cfc608ee3c6e46f3c9edfd60"}
Nov 03 19:45:17.956730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.956705    4531 scope.go:122] "RemoveContainer" containerID="c280a24f54eb0d6fcdeb88bdc6511d8ab2691443cfc608ee3c6e46f3c9edfd60"
Nov 03 19:45:17.957065 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:17.957043    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb_odh-ai-gateway-infra(a398e3b7-2e39-4d05-baa0-1cf429addfc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9"
Nov 03 19:45:17.996462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:17.996426    4531 scope.go:122] "RemoveContainer" containerID="2b10966da9dcb42a9dabedda531035ddadb4c40d27331e0870646fa680b47c8f"
Nov 03 19:45:18.012949 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.012860    4531 scope.go:122] "RemoveContainer" containerID="e4867c29bd2c42e70dab3aa1d45c677c417152f39c2816deb68b344f9a0690c7"
Nov 03 19:45:18.015673 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.015652    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"]
Nov 03 19:45:18.016336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.016321    4531 cpu_manager.go:414] "containerMap: removing container" podUID="716a697b-354d-4e08-baeb-b4a3aed59613" containerName="revoke-keys"
Nov 03 19:45:18.016388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.016337    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716a697b-354d-4e08-baeb-b4a3aed59613" containerName="revoke-keys"
Nov 03 19:45:18.016441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.016432    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="716a697b-354d-4e08-baeb-b4a3aed59613" containerName="revoke-keys"
Nov 03 19:45:18.019989 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.019969    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.027483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.026703    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0486e8-openshift-default-dockercfg-4ckf8\""
Nov 03 19:45:18.035560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.035530    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"]
Nov 03 19:45:18.068694 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.068667    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.068856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.068754    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.068913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.068851    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9542796-9bc6-4afc-b664-35a17397bc61-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.068959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.068920    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9542796-9bc6-4afc-b664-35a17397bc61-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.069107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.069086    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.069157 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.069144    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.069269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.069249    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzxc\" (UniqueName: \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-kube-api-access-qjzxc\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.069317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.069301    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.069435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.069417    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.146857 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.144999    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:18.172200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.171999    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.172200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.172105    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.172200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.172161    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9542796-9bc6-4afc-b664-35a17397bc61-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.172200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.172212    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9542796-9bc6-4afc-b664-35a17397bc61-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.172546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.172322    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.172546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.172375    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.172546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.172420    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzxc\" (UniqueName: \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-kube-api-access-qjzxc\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.172546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.172461    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.172756 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.172571    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.180783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.173115    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.180783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.174676    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9542796-9bc6-4afc-b664-35a17397bc61-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.180783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.175969    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.180783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.176544    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.180783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.176881    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.185196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.181680    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.197767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.197122    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9542796-9bc6-4afc-b664-35a17397bc61-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.197942 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.197841    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.199756 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.199734    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzxc\" (UniqueName: \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-kube-api-access-qjzxc\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.359575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.359542    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:18.645995 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:18.645958    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9542796_9bc6_4afc_b664_35a17397bc61.slice/crio-c14e49a93a6aa4dc50a573de28220ac1726637b7f687f4d53c67185160024952 WatchSource:0}: Error finding container c14e49a93a6aa4dc50a573de28220ac1726637b7f687f4d53c67185160024952: Status 404 returned error can't find the container with id c14e49a93a6aa4dc50a573de28220ac1726637b7f687f4d53c67185160024952
Nov 03 19:45:18.647057 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.646996    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"]
Nov 03 19:45:18.649305 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.649271    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:18.649373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.649336    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:18.649373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.649370    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:18.969321 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.969275    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" event={"ID":"f9542796-9bc6-4afc-b664-35a17397bc61","Type":"ContainerStarted","Data":"d5c899d07753b75ea3fc9beab48520f756fa99113df7f362a333224a1297f2be"}
Nov 03 19:45:18.969791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.969349    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" event={"ID":"f9542796-9bc6-4afc-b664-35a17397bc61","Type":"ContainerStarted","Data":"c14e49a93a6aa4dc50a573de28220ac1726637b7f687f4d53c67185160024952"}
Nov 03 19:45:18.971956 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.971925    4531 generic.go:356] "Generic (PLEG): container finished" podID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerID="7988f0709aad4aa5dfd37fffe90dea197a08254afd3e19e3905aa026afaf76e8" exitCode=60
Nov 03 19:45:18.972070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.971990    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" event={"ID":"f7b2f030-ec61-42c3-b14d-371bbfef3b7a","Type":"ContainerDied","Data":"7988f0709aad4aa5dfd37fffe90dea197a08254afd3e19e3905aa026afaf76e8"}
Nov 03 19:45:18.972070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.972036    4531 scope.go:122] "RemoveContainer" containerID="12a661f89b596aa344c9a6691a78375a628ce3fe01b533471caa2596be18f30a"
Nov 03 19:45:18.972441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.972420    4531 scope.go:122] "RemoveContainer" containerID="7988f0709aad4aa5dfd37fffe90dea197a08254afd3e19e3905aa026afaf76e8"
Nov 03 19:45:18.972779 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:18.972750    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9_odh-ai-gateway-infra(f7b2f030-ec61-42c3-b14d-371bbfef3b7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a"
Nov 03 19:45:18.980996 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.980972    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"]
Nov 03 19:45:18.993600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:18.993543    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podStartSLOduration=1.9935261 podStartE2EDuration="1.9935261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:18.98914208 +0000 UTC m=+832.259265376" watchObservedRunningTime="2026-11-03 19:45:18.9935261 +0000 UTC m=+832.263649397"
Nov 03 19:45:19.008691 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.008657    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"]
Nov 03 19:45:19.061197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.047645    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:19.061197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.053042    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"]
Nov 03 19:45:19.184983 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.184950    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:19.246267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.246234    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"
Nov 03 19:45:19.315027 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.315000    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0a1881-ed0c-488a-b9ef-e029167f5d48-kube-api-access-759pf\" (UniqueName: \"kubernetes.io/projected/cc0a1881-ed0c-488a-b9ef-e029167f5d48-kube-api-access-759pf\") pod \"cc0a1881-ed0c-488a-b9ef-e029167f5d48\" (UID: \"cc0a1881-ed0c-488a-b9ef-e029167f5d48\") "
Nov 03 19:45:19.315196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.315108    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0a1881-ed0c-488a-b9ef-e029167f5d48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0a1881-ed0c-488a-b9ef-e029167f5d48-maas-api-service-ca\") pod \"cc0a1881-ed0c-488a-b9ef-e029167f5d48\" (UID: \"cc0a1881-ed0c-488a-b9ef-e029167f5d48\") "
Nov 03 19:45:19.315947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.315540    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0a1881-ed0c-488a-b9ef-e029167f5d48-maas-api-service-ca" pod "cc0a1881-ed0c-488a-b9ef-e029167f5d48" (UID: "cc0a1881-ed0c-488a-b9ef-e029167f5d48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:19.315947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.315906    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0a1881-ed0c-488a-b9ef-e029167f5d48-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:19.317226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.317199    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0a1881-ed0c-488a-b9ef-e029167f5d48-kube-api-access-759pf" pod "cc0a1881-ed0c-488a-b9ef-e029167f5d48" (UID: "cc0a1881-ed0c-488a-b9ef-e029167f5d48"). InnerVolumeSpecName "kube-api-access-759pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:19.335897 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.335870    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"
Nov 03 19:45:19.339590 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.339568    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"
Nov 03 19:45:19.378230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.360402    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:19.378230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.365324    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:19.378230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.365371    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 19:45:19.422196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.417504    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a59d90-5a71-43f2-a18a-49d29f0123b1-kube-api-access-gqfwp\" (UniqueName: \"kubernetes.io/projected/84a59d90-5a71-43f2-a18a-49d29f0123b1-kube-api-access-gqfwp\") pod \"84a59d90-5a71-43f2-a18a-49d29f0123b1\" (UID: \"84a59d90-5a71-43f2-a18a-49d29f0123b1\") "
Nov 03 19:45:19.422196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.417584    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a398e3b7-2e39-4d05-baa0-1cf429addfc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a398e3b7-2e39-4d05-baa0-1cf429addfc9-maas-api-service-ca\") pod \"a398e3b7-2e39-4d05-baa0-1cf429addfc9\" (UID: \"a398e3b7-2e39-4d05-baa0-1cf429addfc9\") "
Nov 03 19:45:19.422196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.417824    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a59d90-5a71-43f2-a18a-49d29f0123b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a59d90-5a71-43f2-a18a-49d29f0123b1-maas-api-service-ca\") pod \"84a59d90-5a71-43f2-a18a-49d29f0123b1\" (UID: \"84a59d90-5a71-43f2-a18a-49d29f0123b1\") "
Nov 03 19:45:19.422196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.417908    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a398e3b7-2e39-4d05-baa0-1cf429addfc9-kube-api-access-h5799\" (UniqueName: \"kubernetes.io/projected/a398e3b7-2e39-4d05-baa0-1cf429addfc9-kube-api-access-h5799\") pod \"a398e3b7-2e39-4d05-baa0-1cf429addfc9\" (UID: \"a398e3b7-2e39-4d05-baa0-1cf429addfc9\") "
Nov 03 19:45:19.422196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.418773    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-759pf\" (UniqueName: \"kubernetes.io/projected/cc0a1881-ed0c-488a-b9ef-e029167f5d48-kube-api-access-759pf\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:19.422196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.419903    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a398e3b7-2e39-4d05-baa0-1cf429addfc9-maas-api-service-ca" pod "a398e3b7-2e39-4d05-baa0-1cf429addfc9" (UID: "a398e3b7-2e39-4d05-baa0-1cf429addfc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:19.422196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.420380    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a59d90-5a71-43f2-a18a-49d29f0123b1-maas-api-service-ca" pod "84a59d90-5a71-43f2-a18a-49d29f0123b1" (UID: "84a59d90-5a71-43f2-a18a-49d29f0123b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:19.430237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.429961    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a398e3b7-2e39-4d05-baa0-1cf429addfc9-kube-api-access-h5799" pod "a398e3b7-2e39-4d05-baa0-1cf429addfc9" (UID: "a398e3b7-2e39-4d05-baa0-1cf429addfc9"). InnerVolumeSpecName "kube-api-access-h5799". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:19.431409 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.431377    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a59d90-5a71-43f2-a18a-49d29f0123b1-kube-api-access-gqfwp" pod "84a59d90-5a71-43f2-a18a-49d29f0123b1" (UID: "84a59d90-5a71-43f2-a18a-49d29f0123b1"). InnerVolumeSpecName "kube-api-access-gqfwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:19.519814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.519718    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a59d90-5a71-43f2-a18a-49d29f0123b1-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:19.519814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.519752    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5799\" (UniqueName: \"kubernetes.io/projected/a398e3b7-2e39-4d05-baa0-1cf429addfc9-kube-api-access-h5799\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:19.519814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.519766    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqfwp\" (UniqueName: \"kubernetes.io/projected/84a59d90-5a71-43f2-a18a-49d29f0123b1-kube-api-access-gqfwp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:19.519814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.519778    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a398e3b7-2e39-4d05-baa0-1cf429addfc9-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:19.786253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.785616    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:19.986370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.986333    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"
Nov 03 19:45:19.986914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.986342    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch" event={"ID":"84a59d90-5a71-43f2-a18a-49d29f0123b1","Type":"ContainerDied","Data":"c6d7745a698ada3a5b769c53d2ed9cd29213890906c2f783a1e959896da2d25a"}
Nov 03 19:45:19.986914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.986485    4531 scope.go:122] "RemoveContainer" containerID="3e91efdb41e4d036161e4a5f14b2ae556f7b543c783f45086aba6fae8d6426d2"
Nov 03 19:45:19.993574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.993551    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"
Nov 03 19:45:19.993574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:19.993560    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb" event={"ID":"a398e3b7-2e39-4d05-baa0-1cf429addfc9","Type":"ContainerDied","Data":"4bef1555ccddc4db8c12516459ea9ba30d6d3f9c8660150d669f5ea97d004a5b"}
Nov 03 19:45:20.000255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.000200    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"
Nov 03 19:45:20.000380 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.000200    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp" event={"ID":"cc0a1881-ed0c-488a-b9ef-e029167f5d48","Type":"ContainerDied","Data":"795c29e11526e18c508027e32adbf37e5bc839563c13c9653cc14e4cd020a055"}
Nov 03 19:45:20.001332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.001294    4531 scope.go:122] "RemoveContainer" containerID="c280a24f54eb0d6fcdeb88bdc6511d8ab2691443cfc608ee3c6e46f3c9edfd60"
Nov 03 19:45:20.011912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.011894    4531 scope.go:122] "RemoveContainer" containerID="dc401d5505ace029f6009926388a489721314e541d3389404d0205187a814a64"
Nov 03 19:45:20.014521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.014495    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"]
Nov 03 19:45:20.021099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.021081    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895585-hc9ch"]
Nov 03 19:45:20.025831 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.025810    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"]
Nov 03 19:45:20.067251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.062865    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"]
Nov 03 19:45:20.074707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.074671    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895585-t64qb"]
Nov 03 19:45:20.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.081586    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"]
Nov 03 19:45:20.086624 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.086599    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895585-nrdnp"]
Nov 03 19:45:20.231209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.231162    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"
Nov 03 19:45:20.331833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.331758    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-kube-api-access-ffp52\" (UniqueName: \"kubernetes.io/projected/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-kube-api-access-ffp52\") pod \"f7b2f030-ec61-42c3-b14d-371bbfef3b7a\" (UID: \"f7b2f030-ec61-42c3-b14d-371bbfef3b7a\") "
Nov 03 19:45:20.331833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.331822    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-maas-api-service-ca\") pod \"f7b2f030-ec61-42c3-b14d-371bbfef3b7a\" (UID: \"f7b2f030-ec61-42c3-b14d-371bbfef3b7a\") "
Nov 03 19:45:20.332163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.332143    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-maas-api-service-ca" pod "f7b2f030-ec61-42c3-b14d-371bbfef3b7a" (UID: "f7b2f030-ec61-42c3-b14d-371bbfef3b7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:20.332423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.332408    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:20.334015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.333987    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-kube-api-access-ffp52" pod "f7b2f030-ec61-42c3-b14d-371bbfef3b7a" (UID: "f7b2f030-ec61-42c3-b14d-371bbfef3b7a"). InnerVolumeSpecName "kube-api-access-ffp52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:20.334330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.334307    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:20.362321 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.362288    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:20.362471 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.362349    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 19:45:20.433925 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.433891    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffp52\" (UniqueName: \"kubernetes.io/projected/f7b2f030-ec61-42c3-b14d-371bbfef3b7a-kube-api-access-ffp52\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:20.465024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.464994    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:20.598705 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:20.598615    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:45:20.609149 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.609123    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:20.870339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.870264    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1" path="/var/lib/kubelet/pods/84a59d90-5a71-43f2-a18a-49d29f0123b1/volumes"
Nov 03 19:45:20.870800 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.870785    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" path="/var/lib/kubelet/pods/a398e3b7-2e39-4d05-baa0-1cf429addfc9/volumes"
Nov 03 19:45:20.871271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:20.871250    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" path="/var/lib/kubelet/pods/cc0a1881-ed0c-488a-b9ef-e029167f5d48/volumes"
Nov 03 19:45:21.013284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:21.013255    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"
Nov 03 19:45:21.013284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:21.013263    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9" event={"ID":"f7b2f030-ec61-42c3-b14d-371bbfef3b7a","Type":"ContainerDied","Data":"f6f2b2cbad19a8843b2205680b50a30d2e61e9045161ee916b89a416e5c58d20"}
Nov 03 19:45:21.013835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:21.013333    4531 scope.go:122] "RemoveContainer" containerID="7988f0709aad4aa5dfd37fffe90dea197a08254afd3e19e3905aa026afaf76e8"
Nov 03 19:45:21.033544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:21.033520    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"]
Nov 03 19:45:21.061319 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:21.050127    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895585-9xzk9"]
Nov 03 19:45:21.361884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:21.361847    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:21.362106 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:21.361910    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 19:45:21.516201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:21.515750    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:22.331803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.331767    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:22.367197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.366941    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:22.367197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.367031    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 19:45:22.868422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.868387    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" path="/var/lib/kubelet/pods/f7b2f030-ec61-42c3-b14d-371bbfef3b7a/volumes"
Nov 03 19:45:22.994186 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994131    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"]
Nov 03 19:45:22.994844 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994822    4531 cpu_manager.go:414] "containerMap: removing container" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerName="cleanup"
Nov 03 19:45:22.994844 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994844    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994862    4531 cpu_manager.go:414] "containerMap: removing container" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994870    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994886    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994894    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994910    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994917    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994946    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994953    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994967    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994977    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.994993    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerName="cleanup"
Nov 03 19:45:22.995002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995001    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995018    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995026    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995045    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995052    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995086    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995094    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995262    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995278    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995287    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995297    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995312    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995324    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995335    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995347    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerName="cleanup"
Nov 03 19:45:22.995588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:22.995355    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerName="cleanup"
Nov 03 19:45:23.001269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.001244    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:23.004417 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.004390    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:45:23.004547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.004428    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:45:23.004793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.004766    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"]
Nov 03 19:45:23.107101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.107069    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3013387d-fqvsk\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:23.107296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.107206    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6t6m\" (UniqueName: \"kubernetes.io/projected/a877371b-020a-44fd-b07b-80bd5a58c75a-kube-api-access-s6t6m\") pod \"maas-api-revoke-sub-3013387d-fqvsk\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:23.107296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.107262    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3013387d-fqvsk\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:23.173195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.169634    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:23.216299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.211234    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3013387d-fqvsk\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:23.216299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.211370    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6t6m\" (UniqueName: \"kubernetes.io/projected/a877371b-020a-44fd-b07b-80bd5a58c75a-kube-api-access-s6t6m\") pod \"maas-api-revoke-sub-3013387d-fqvsk\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:23.216299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.211450    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3013387d-fqvsk\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:23.216299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.212404    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3013387d-fqvsk\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:23.226687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.226526    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3013387d-fqvsk\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:23.227937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.227913    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6t6m\" (UniqueName: \"kubernetes.io/projected/a877371b-020a-44fd-b07b-80bd5a58c75a-kube-api-access-s6t6m\") pod \"maas-api-revoke-sub-3013387d-fqvsk\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:23.322802 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.322767    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:23.363417 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.362703    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:23.363417 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.362830    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 19:45:23.661210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:23.657149    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"]
Nov 03 19:45:23.661746 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:23.661710    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda877371b_020a_44fd_b07b_80bd5a58c75a.slice/crio-b976c06553bcf626e14c69a6622b725215233505c0e8a11a8181860debca5ef8 WatchSource:0}: Error finding container b976c06553bcf626e14c69a6622b725215233505c0e8a11a8181860debca5ef8: Status 404 returned error can't find the container with id b976c06553bcf626e14c69a6622b725215233505c0e8a11a8181860debca5ef8
Nov 03 19:45:24.054196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:24.049417    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk" event={"ID":"a877371b-020a-44fd-b07b-80bd5a58c75a","Type":"ContainerStarted","Data":"b976c06553bcf626e14c69a6622b725215233505c0e8a11a8181860debca5ef8"}
Nov 03 19:45:24.119656 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:24.119621    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:24.369250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:24.361310    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:24.369250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:24.361374    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 19:45:25.164205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.164084    4531 generic.go:356] "Generic (PLEG): container finished" podID="a877371b-020a-44fd-b07b-80bd5a58c75a" containerID="1dbc92c575684bc26c9d26493ce48c5801264285202f9d38fd50d2f71ef27052" exitCode=0
Nov 03 19:45:25.164560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.164519    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk" event={"ID":"a877371b-020a-44fd-b07b-80bd5a58c75a","Type":"ContainerDied","Data":"1dbc92c575684bc26c9d26493ce48c5801264285202f9d38fd50d2f71ef27052"}
Nov 03 19:45:25.317411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.316553    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:25.359316 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.358714    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:25.360340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.360249    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:25.360340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.360293    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 19:45:25.369850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.369791    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:45:25.484195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.484070    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:25.582592 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.582547    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:25.629196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.619341    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:25.714014 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.713958    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:25.762260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.739664    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:45:25.762260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.741808    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:25.804220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.791744    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:25.828218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.818262    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:25.888776 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.869276    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:25.944067 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:25.944001    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.052267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.019418    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.327257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.326273    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.327257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.327139    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:45:26.363198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.362413    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.363198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.363130    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:45:26.375198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.364908    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:26.375198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.364949    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 19:45:26.405707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.396254    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.405707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.397356    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:45:26.405707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.400149    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.498334 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.485549    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.572859 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.506642    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:45:26.687442 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.589270    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.687442 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.618862    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:45:26.687442 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.656478    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:45:26.687442 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.656801    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:45:26.743496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.743371    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:45:26.778810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.745701    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:45:26.778810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.746931    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:45:26.778810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.776405    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:45:26.892237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.873606    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.892237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.873720    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:45:26.892237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.877357    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:45:26.971305 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.902063    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:45:27.026219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:26.984287    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.201382 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.091250    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:45:27.233335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.230111    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:27.233335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.230190    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.438201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.335907    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.502308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.369297    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:45:27.502308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.369343    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:27.502308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.453385    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.502308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.397580    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.590224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.537158    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:45:27.590224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.561921    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:27.590224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.561973    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 19:45:27.590224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.566683    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.592062    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.593745    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.595706    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.595767    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.644714    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.644784    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.659257    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.676647    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.701320    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.703292    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.703355    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.768001    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.801477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:27.768923    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:45:28.043218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.020317    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:28.253197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.225720    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:28.253197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.227813    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:45:28.253197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.227886    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:45:28.311411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.297446    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:28.556955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.556901    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:28.591765 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.576067    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:45:28.601477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.601436    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:28.602295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.602264    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:45:28.602961 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.602937    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:45:28.631196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.631154    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:45:28.645803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.645734    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": context deadline exceeded"
Nov 03 19:45:28.781253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.780669    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:28.781253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.780731    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:28.818511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.811253    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Nov 03 19:45:28.887784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:28.886845    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:45:29.197301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.197255    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:45:29.234548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.201810    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:29.234855 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.207147    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:45:29.235016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.228623    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:29.235097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.233756    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:29.246298 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.246275    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:45:29.246526 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.246510    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:45:29.246677 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.246663    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:45:29.246804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.246794    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:45:29.248856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.248835    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:45:29.251223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.251206    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:45:29.257553 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.257537    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:45:29.262478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.262462    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:45:29.271208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.266583    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:45:29.280466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.280428    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"]
Nov 03 19:45:29.282064 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.282042    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a877371b-020a-44fd-b07b-80bd5a58c75a" containerName="revoke-keys"
Nov 03 19:45:29.282317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.282303    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a877371b-020a-44fd-b07b-80bd5a58c75a" containerName="revoke-keys"
Nov 03 19:45:29.282430 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.282419    4531 cpu_manager.go:414] "containerMap: removing container" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerName="cleanup"
Nov 03 19:45:29.282519 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.282508    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a59d90-5a71-43f2-a18a-49d29f0123b1" containerName="cleanup"
Nov 03 19:45:29.282620 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.282608    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerName="cleanup"
Nov 03 19:45:29.282700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.282690    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerName="cleanup"
Nov 03 19:45:29.283000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.282968    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b2f030-ec61-42c3-b14d-371bbfef3b7a" containerName="cleanup"
Nov 03 19:45:29.283114 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.283097    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a398e3b7-2e39-4d05-baa0-1cf429addfc9" containerName="cleanup"
Nov 03 19:45:29.283250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.283240    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0a1881-ed0c-488a-b9ef-e029167f5d48" containerName="cleanup"
Nov 03 19:45:29.283366 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.283356    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a877371b-020a-44fd-b07b-80bd5a58c75a" containerName="revoke-keys"
Nov 03 19:45:29.287277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.287262    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"]
Nov 03 19:45:29.287438 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.287428    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:29.344238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.344210    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-openshift-ca\") pod \"a877371b-020a-44fd-b07b-80bd5a58c75a\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") "
Nov 03 19:45:29.344829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.344707    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-openshift-ca" pod "a877371b-020a-44fd-b07b-80bd5a58c75a" (UID: "a877371b-020a-44fd-b07b-80bd5a58c75a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:29.344829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.344798    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-xks-ca\") pod \"a877371b-020a-44fd-b07b-80bd5a58c75a\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") "
Nov 03 19:45:29.345007 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.344943    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a877371b-020a-44fd-b07b-80bd5a58c75a-kube-api-access-s6t6m\" (UniqueName: \"kubernetes.io/projected/a877371b-020a-44fd-b07b-80bd5a58c75a-kube-api-access-s6t6m\") pod \"a877371b-020a-44fd-b07b-80bd5a58c75a\" (UID: \"a877371b-020a-44fd-b07b-80bd5a58c75a\") "
Nov 03 19:45:29.345334 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.345308    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24844a64-5ftq8\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:29.345420 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.345374    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlr9h\" (UniqueName: \"kubernetes.io/projected/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-kube-api-access-dlr9h\") pod \"maas-api-revoke-sub-24844a64-5ftq8\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:29.345526 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.345507    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24844a64-5ftq8\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:29.347552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.347403    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:29.355679 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.355632    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a877371b-020a-44fd-b07b-80bd5a58c75a-kube-api-access-s6t6m" pod "a877371b-020a-44fd-b07b-80bd5a58c75a" (UID: "a877371b-020a-44fd-b07b-80bd5a58c75a"). InnerVolumeSpecName "kube-api-access-s6t6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:29.362910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.362067    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-xks-ca" pod "a877371b-020a-44fd-b07b-80bd5a58c75a" (UID: "a877371b-020a-44fd-b07b-80bd5a58c75a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:29.377324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.372787    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:29.377324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.373290    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:29.377324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.374967    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 19:45:29.449297 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.449164    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24844a64-5ftq8\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:29.449465 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.449295    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlr9h\" (UniqueName: \"kubernetes.io/projected/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-kube-api-access-dlr9h\") pod \"maas-api-revoke-sub-24844a64-5ftq8\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:29.449465 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.449404    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24844a64-5ftq8\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:29.449938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.449916    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a877371b-020a-44fd-b07b-80bd5a58c75a-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:29.450043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.449944    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6t6m\" (UniqueName: \"kubernetes.io/projected/a877371b-020a-44fd-b07b-80bd5a58c75a-kube-api-access-s6t6m\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:29.451332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.451302    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24844a64-5ftq8\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:29.465408 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.460619    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24844a64-5ftq8\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:29.466804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.466775    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlr9h\" (UniqueName: \"kubernetes.io/projected/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-kube-api-access-dlr9h\") pod \"maas-api-revoke-sub-24844a64-5ftq8\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:29.606946 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.606916    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:29.895477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.895435    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:29.936250 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:29.936165    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e92b7d4_cc64_44ea_99b3_1b3db7473f7c.slice/crio-1d342f6d193f83eb7ed8f138832142c3f11262ee361ad566b59a3cff1f95d4a0 WatchSource:0}: Error finding container 1d342f6d193f83eb7ed8f138832142c3f11262ee361ad566b59a3cff1f95d4a0: Status 404 returned error can't find the container with id 1d342f6d193f83eb7ed8f138832142c3f11262ee361ad566b59a3cff1f95d4a0
Nov 03 19:45:29.937508 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.937475    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk" event={"ID":"a877371b-020a-44fd-b07b-80bd5a58c75a","Type":"ContainerDied","Data":"b976c06553bcf626e14c69a6622b725215233505c0e8a11a8181860debca5ef8"}
Nov 03 19:45:29.937705 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.937686    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b976c06553bcf626e14c69a6622b725215233505c0e8a11a8181860debca5ef8"
Nov 03 19:45:29.938834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.938633    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"
Nov 03 19:45:29.965951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:29.964575    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"]
Nov 03 19:45:30.641199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:30.635728    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:30.946227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:30.946119    4531 generic.go:356] "Generic (PLEG): container finished" podID="5e92b7d4-cc64-44ea-99b3-1b3db7473f7c" containerID="b2d39f64192d294c8e8ff426f2ec83a9609e545648636684278cd16a16d85226" exitCode=0
Nov 03 19:45:30.946227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:30.946212    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8" event={"ID":"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c","Type":"ContainerDied","Data":"b2d39f64192d294c8e8ff426f2ec83a9609e545648636684278cd16a16d85226"}
Nov 03 19:45:30.946415 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:30.946264    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8" event={"ID":"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c","Type":"ContainerStarted","Data":"1d342f6d193f83eb7ed8f138832142c3f11262ee361ad566b59a3cff1f95d4a0"}
Nov 03 19:45:31.383248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:31.379687    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:32.021483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.021441    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:32.164408 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.164384    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:32.314383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.314307    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-kube-api-access-dlr9h\" (UniqueName: \"kubernetes.io/projected/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-kube-api-access-dlr9h\") pod \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") "
Nov 03 19:45:32.314530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.314434    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-openshift-ca\") pod \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") "
Nov 03 19:45:32.314530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.314456    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-xks-ca\") pod \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\" (UID: \"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c\") "
Nov 03 19:45:32.317044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.315256    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-openshift-ca" pod "5e92b7d4-cc64-44ea-99b3-1b3db7473f7c" (UID: "5e92b7d4-cc64-44ea-99b3-1b3db7473f7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:32.317044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.315821    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:32.317924 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.317891    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-kube-api-access-dlr9h" pod "5e92b7d4-cc64-44ea-99b3-1b3db7473f7c" (UID: "5e92b7d4-cc64-44ea-99b3-1b3db7473f7c"). InnerVolumeSpecName "kube-api-access-dlr9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:32.326490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.326460    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-xks-ca" pod "5e92b7d4-cc64-44ea-99b3-1b3db7473f7c" (UID: "5e92b7d4-cc64-44ea-99b3-1b3db7473f7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:32.417564 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.417529    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlr9h\" (UniqueName: \"kubernetes.io/projected/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-kube-api-access-dlr9h\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:32.417564 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.417564    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:32.632433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.632401    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"]
Nov 03 19:45:32.633158 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.633139    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5e92b7d4-cc64-44ea-99b3-1b3db7473f7c" containerName="revoke-keys"
Nov 03 19:45:32.633320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.633165    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e92b7d4-cc64-44ea-99b3-1b3db7473f7c" containerName="revoke-keys"
Nov 03 19:45:32.633378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.633350    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e92b7d4-cc64-44ea-99b3-1b3db7473f7c" containerName="revoke-keys"
Nov 03 19:45:32.637059 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.637039    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.647717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.647696    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"]
Nov 03 19:45:32.721476 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.721446    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5j2b\" (UniqueName: \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-kube-api-access-d5j2b\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.721628 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.721516    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.721628 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.721607    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.721703 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.721661    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.721774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.721761    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02097b90-294d-4de0-bb01-c4bb50228769-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.721856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.721844    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.721892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.721867    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02097b90-294d-4de0-bb01-c4bb50228769-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.721988 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.721966    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.722021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.722001    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.823620 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.823591    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.823620 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.823622    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02097b90-294d-4de0-bb01-c4bb50228769-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.823816 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.823684    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.823816 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.823702    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.823816 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.823741    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5j2b\" (UniqueName: \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-kube-api-access-d5j2b\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.823816 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.823802    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.823990 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.823825    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.824045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.823978    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.824092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.824046    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.824169 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.824139    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02097b90-294d-4de0-bb01-c4bb50228769-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.824362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.824335    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.824432 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.824340    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.824493 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.824423    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.824746 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.824723    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02097b90-294d-4de0-bb01-c4bb50228769-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.826135 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.826115    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.826372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.826354    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02097b90-294d-4de0-bb01-c4bb50228769-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.830371 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.830351    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.830708 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.830691    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5j2b\" (UniqueName: \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-kube-api-access-d5j2b\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.905538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.903849    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:32.956119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.956087    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:32.964789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.964761    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8" event={"ID":"5e92b7d4-cc64-44ea-99b3-1b3db7473f7c","Type":"ContainerDied","Data":"1d342f6d193f83eb7ed8f138832142c3f11262ee361ad566b59a3cff1f95d4a0"}
Nov 03 19:45:32.964904 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.964817    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d342f6d193f83eb7ed8f138832142c3f11262ee361ad566b59a3cff1f95d4a0"
Nov 03 19:45:32.964904 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:32.964885    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"
Nov 03 19:45:33.051090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.051054    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"]
Nov 03 19:45:33.057264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.057240    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.072877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.072129    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"]
Nov 03 19:45:33.129183 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.129150    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.129324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.129257    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.129324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.129317    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.129437 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.129425    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.129561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.129537    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.129613 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.129594    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.129645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.129633    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.129676 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.129649    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.129706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.129683    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkqs\" (UniqueName: \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-kube-api-access-9fkqs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.230853 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.230769    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.230853 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.230831    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.231075 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.230859    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.231075 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.230885    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkqs\" (UniqueName: \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-kube-api-access-9fkqs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.231075 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.231047    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.231404 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.231077    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.231531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.231456    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.231586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.231352    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.231586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.231387    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.231586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.231575    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.231755 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.231124    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.231755 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.231724    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.232064 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.232045    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.232286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.232258    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.233345 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.233322    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.233868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.233847    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.239763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.239742    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkqs\" (UniqueName: \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-kube-api-access-9fkqs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.239964 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.239944    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.249209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.249184    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"]
Nov 03 19:45:33.256387 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:33.256357    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02097b90_294d_4de0_bb01_c4bb50228769.slice/crio-fd84c1e487a3a309c9847cbbb5eeaf571b18da9263ba02479463061ab37f78c2 WatchSource:0}: Error finding container fd84c1e487a3a309c9847cbbb5eeaf571b18da9263ba02479463061ab37f78c2: Status 404 returned error can't find the container with id fd84c1e487a3a309c9847cbbb5eeaf571b18da9263ba02479463061ab37f78c2
Nov 03 19:45:33.259949 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.259919    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:33.260017 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.259984    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:33.260063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.260016    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:33.383357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.383326    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:33.678336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.678303    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"]
Nov 03 19:45:33.679757 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:33.679724    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebdbfae_b6b0_4ea1_a180_29de52d1cc60.slice/crio-6a86d7e06788eb686cf70ae9bd44bf4f3af4a3acdb5260544d41972cd9df4b43 WatchSource:0}: Error finding container 6a86d7e06788eb686cf70ae9bd44bf4f3af4a3acdb5260544d41972cd9df4b43: Status 404 returned error can't find the container with id 6a86d7e06788eb686cf70ae9bd44bf4f3af4a3acdb5260544d41972cd9df4b43
Nov 03 19:45:33.682965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.682929    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:33.683041 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.683003    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:33.683096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.683038    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:33.712083 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.712037    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:33.987732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.987640    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" event={"ID":"bebdbfae-b6b0-4ea1-a180-29de52d1cc60","Type":"ContainerStarted","Data":"748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9"}
Nov 03 19:45:33.987732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.987704    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" event={"ID":"bebdbfae-b6b0-4ea1-a180-29de52d1cc60","Type":"ContainerStarted","Data":"6a86d7e06788eb686cf70ae9bd44bf4f3af4a3acdb5260544d41972cd9df4b43"}
Nov 03 19:45:33.990694 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.990664    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" event={"ID":"02097b90-294d-4de0-bb01-c4bb50228769","Type":"ContainerStarted","Data":"50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b"}
Nov 03 19:45:33.990834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:33.990712    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" event={"ID":"02097b90-294d-4de0-bb01-c4bb50228769","Type":"ContainerStarted","Data":"fd84c1e487a3a309c9847cbbb5eeaf571b18da9263ba02479463061ab37f78c2"}
Nov 03 19:45:34.076229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:34.074382    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" podStartSLOduration=1.074364046 podStartE2EDuration="1.074364046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:34.045555118 +0000 UTC m=+847.315678416" watchObservedRunningTime="2026-11-03 19:45:34.074364046 +0000 UTC m=+847.344487341"
Nov 03 19:45:34.101006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:34.100952    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" podStartSLOduration=2.100939157 podStartE2EDuration="2.100939157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:34.098853963 +0000 UTC m=+847.368977257" watchObservedRunningTime="2026-11-03 19:45:34.100939157 +0000 UTC m=+847.371062511"
Nov 03 19:45:34.390197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:34.383785    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:34.390197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:34.385710    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:34.390197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:34.385777    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 19:45:34.795817 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:34.795715    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:34.957567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:34.957533    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:34.964328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:34.964296    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 19:45:34.964562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:34.964535    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" podUID="02097b90-294d-4de0-bb01-c4bb50228769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 19:45:35.384684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:35.384610    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:35.385207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:35.384709    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 19:45:35.671853 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:35.671082    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:35.957257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:35.956589    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 19:45:35.957257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:35.956669    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" podUID="02097b90-294d-4de0-bb01-c4bb50228769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 19:45:36.385261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:36.384675    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:36.385261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:36.384749    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 19:45:36.665368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:36.665164    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:36.957644 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:36.957553    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 19:45:36.958327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:36.958295    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" podUID="02097b90-294d-4de0-bb01-c4bb50228769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 19:45:37.385606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:37.385568    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:37.386597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:37.386150    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 19:45:37.546205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:37.545717    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:37.957600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:37.957560    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 19:45:37.979348 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:37.979297    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" podUID="02097b90-294d-4de0-bb01-c4bb50228769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 19:45:38.389248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:38.389131    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:38.389248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:38.389229    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 19:45:38.665256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:38.665137    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:38.957251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:38.956555    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 19:45:38.957251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:38.956620    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" podUID="02097b90-294d-4de0-bb01-c4bb50228769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 19:45:39.384047 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:39.384012    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:39.384248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:39.384082    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 19:45:39.622032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:39.621940    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:39.956780 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:39.956742    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 19:45:39.956992 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:39.956816    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" podUID="02097b90-294d-4de0-bb01-c4bb50228769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 19:45:40.357256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:40.351655    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:40.386034 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:40.385994    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:40.386622 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:40.386322    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 19:45:40.961864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:40.961828    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:40.962300 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:40.962285    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:40.962939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:40.962915    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:45:41.236004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:41.235896    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:41.404851 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:41.404820    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:41.405231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:41.405166    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:41.410582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:41.406630    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:45:42.076196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:42.071828    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:42.849199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:42.845010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:43.739197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:43.734336    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:44.464864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:44.464825    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:45.150758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.150724    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:45.423018 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.422945    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"]
Nov 03 19:45:45.427028 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.427010    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:45.429932 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.429904    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:45:45.430076 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.430019    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:45:45.435764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.435733    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"]
Nov 03 19:45:45.529553 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.529525    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268e9995-sgxjv\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:45.529705 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.529572    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmgf7\" (UniqueName: \"kubernetes.io/projected/970b169f-9ebf-4fed-b9d1-44ee2f761c58-kube-api-access-kmgf7\") pod \"maas-api-revoke-sub-268e9995-sgxjv\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:45.529747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.529737    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268e9995-sgxjv\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:45.631659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.631628    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268e9995-sgxjv\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:45.631843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.631824    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268e9995-sgxjv\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:45.631897 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.631882    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmgf7\" (UniqueName: \"kubernetes.io/projected/970b169f-9ebf-4fed-b9d1-44ee2f761c58-kube-api-access-kmgf7\") pod \"maas-api-revoke-sub-268e9995-sgxjv\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:45.632424 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.632403    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268e9995-sgxjv\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:45.634244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.634222    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268e9995-sgxjv\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:45.638709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.638684    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmgf7\" (UniqueName: \"kubernetes.io/projected/970b169f-9ebf-4fed-b9d1-44ee2f761c58-kube-api-access-kmgf7\") pod \"maas-api-revoke-sub-268e9995-sgxjv\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:45.744711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:45.744643    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:46.020236 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:46.020202    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970b169f_9ebf_4fed_b9d1_44ee2f761c58.slice/crio-a1f4e5866d0728627e0e06808674541bea4896086d00178ead92fb0b22fb0d59 WatchSource:0}: Error finding container a1f4e5866d0728627e0e06808674541bea4896086d00178ead92fb0b22fb0d59: Status 404 returned error can't find the container with id a1f4e5866d0728627e0e06808674541bea4896086d00178ead92fb0b22fb0d59
Nov 03 19:45:46.020495 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:46.020469    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"]
Nov 03 19:45:46.038721 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:46.038689    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:46.191377 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:46.191344    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv" event={"ID":"970b169f-9ebf-4fed-b9d1-44ee2f761c58","Type":"ContainerStarted","Data":"a1f4e5866d0728627e0e06808674541bea4896086d00178ead92fb0b22fb0d59"}
Nov 03 19:45:46.745840 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:46.745762    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:47.199800 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.199764    4531 generic.go:356] "Generic (PLEG): container finished" podID="970b169f-9ebf-4fed-b9d1-44ee2f761c58" containerID="c22985759731f58419dee6dc1ba1293eeb027fa8506c6c35310114d43eb2d1b2" exitCode=0
Nov 03 19:45:47.200200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.199805    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv" event={"ID":"970b169f-9ebf-4fed-b9d1-44ee2f761c58","Type":"ContainerDied","Data":"c22985759731f58419dee6dc1ba1293eeb027fa8506c6c35310114d43eb2d1b2"}
Nov 03 19:45:47.485375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.485334    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:47.565756 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.565722    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"]
Nov 03 19:45:47.570356 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.570332    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.583129 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.583106    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"]
Nov 03 19:45:47.675545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.675510    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"]
Nov 03 19:45:47.676082 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.676054    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.676198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.676100    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.676313 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.676275    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.676442 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.676390    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.676514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.676458    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e49e9f-167a-441d-824b-d8499196e409-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.676514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.676490    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.676619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.676600    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e49e9f-167a-441d-824b-d8499196e409-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.676699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.676680    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.676848 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.676828    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4v4b\" (UniqueName: \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-kube-api-access-t4v4b\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.681016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.680992    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.690732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.690684    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"]
Nov 03 19:45:47.734820 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.734778    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"]
Nov 03 19:45:47.742038 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.741972    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.742281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.742255    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"]
Nov 03 19:45:47.778046 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778020    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.778201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778062    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.778201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778121    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.778201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778155    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.778367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778256    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.778425 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778371    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e49e9f-167a-441d-824b-d8499196e409-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.778425 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778412    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.778535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778439    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.778535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778474    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbhr\" (UniqueName: \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-kube-api-access-wdbhr\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.778535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778536    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.778706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778576    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.778706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778662    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.778706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778689    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.778834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778735    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.778834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778768    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.778834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.778802    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e49e9f-167a-441d-824b-d8499196e409-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.779052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.779031    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.779125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.779102    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.779193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.779131    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v4b\" (UniqueName: \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-kube-api-access-t4v4b\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.779292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.779251    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.779502 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.779477    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.779502 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.779383    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e49e9f-167a-441d-824b-d8499196e409-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.779663 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.779491    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.780604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.780585    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.781459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.781437    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e49e9f-167a-441d-824b-d8499196e409-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.787705 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.787681    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.788463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.788441    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4v4b\" (UniqueName: \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-kube-api-access-t4v4b\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.881107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881070    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.881306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881114    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.881306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881242    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.881411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881340    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbhr\" (UniqueName: \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-kube-api-access-wdbhr\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.881411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881386    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.881411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881406    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.881539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881436    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfl6h\" (UniqueName: \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-kube-api-access-hfl6h\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.881539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881519    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.881648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881562    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.881648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881594    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.881648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881630    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.881793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881673    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9883aa-fca8-416e-8667-85354d4c2702-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.881793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881777    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.881874 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881845    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.881955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.881920    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.882200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.882058    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.882200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.882081    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.882200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.882109    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9883aa-fca8-416e-8667-85354d4c2702-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.882200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.882127    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.882483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.882219    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.882483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.882276    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.882730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.882702    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.882768 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.882735    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.884399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.884381    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.884635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.884614    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.885478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.885458    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:47.892071 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.892047    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbhr\" (UniqueName: \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-kube-api-access-wdbhr\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.892374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.892358    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:47.939923 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.939894    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"]
Nov 03 19:45:47.953901 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.953864    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:47.957004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.956962    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"]
Nov 03 19:45:47.983934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.983902    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.984124 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.983963    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfl6h\" (UniqueName: \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-kube-api-access-hfl6h\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.984124 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.984029    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9883aa-fca8-416e-8667-85354d4c2702-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.984353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.984218    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.984353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.984277    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.984353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.984310    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9883aa-fca8-416e-8667-85354d4c2702-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.984582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.984415    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.984582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.984449    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.984582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.984504    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.984976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.984952    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.986807 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.986315    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.986807 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.986605    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.988218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.988033    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.990238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.990217    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9883aa-fca8-416e-8667-85354d4c2702-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.993087 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.993018    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9883aa-fca8-416e-8667-85354d4c2702-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.995987 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.995942    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfl6h\" (UniqueName: \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-kube-api-access-hfl6h\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:47.996750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:47.996704    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:48.002567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.001454    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:48.008469 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.008452    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:48.059682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.059441    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:48.085730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.085701    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645da40a-5b03-4200-adba-b2da577b27f3-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.085838 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.085738    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645da40a-5b03-4200-adba-b2da577b27f3-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.085998 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.085977    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.086144 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.086124    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.086310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.086290    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.086439 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.086424    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.086496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.086483    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8j75\" (UniqueName: \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-kube-api-access-z8j75\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.086560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.086551    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.086622 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.086574    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.185931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.185758    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"]
Nov 03 19:45:48.188401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.188367    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.188548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.188422    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.188548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.188472    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645da40a-5b03-4200-adba-b2da577b27f3-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.188548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.188499    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645da40a-5b03-4200-adba-b2da577b27f3-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.192004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.188629    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.192004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.189304    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.192004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.189352    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.192004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.189457    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.192004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.189532    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8j75\" (UniqueName: \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-kube-api-access-z8j75\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.192004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.190445    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.192004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.191758    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645da40a-5b03-4200-adba-b2da577b27f3-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.192436 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.192416    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.193835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.192780    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"]
Nov 03 19:45:48.193835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.192917    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.193835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.193151    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.196859 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.194627    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.196859 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.196104    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.196859 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.196165    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.196859 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.196210    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.197102 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.197009    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645da40a-5b03-4200-adba-b2da577b27f3-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.298227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.290092    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8j75\" (UniqueName: \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-kube-api-access-z8j75\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.298227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.296429    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"]
Nov 03 19:45:48.298227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.296719    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.323645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.323611    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:48.362754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.330104    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" event={"ID":"c6e49e9f-167a-441d-824b-d8499196e409","Type":"ContainerStarted","Data":"5208b4609f59926f0da55e14e747b33d2e4579c85b9eb08ed04968735e999e47"}
Nov 03 19:45:48.364624 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.364588    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.365068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.365023    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.365351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.365297    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.365495 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.365474    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.375504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.369975    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"]
Nov 03 19:45:48.383886 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.383833    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"]
Nov 03 19:45:48.385274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.385235    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.385274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.385320    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.385274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.385363    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.399753 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.399717    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.399914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.399794    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.399914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.399858    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.400048 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.400023    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.400257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.400215    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.400368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.400350    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65jcw\" (UniqueName: \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-kube-api-access-65jcw\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.400434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.400409    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.400510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.400495    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.400615 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.400573    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.505125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.503287    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.505125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.503389    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65jcw\" (UniqueName: \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-kube-api-access-65jcw\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.505125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.503424    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.505125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.503489    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.505125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.503548    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.505125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.503605    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.505125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.503644    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.505125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.503674    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.505125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.503810    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.508384 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.507601    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.508384 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.508083    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.509191 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.509153    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.509586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.509565    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.511003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.510922    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.511003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.510980    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.519724 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.519689    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.543362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.543042    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65jcw\" (UniqueName: \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-kube-api-access-65jcw\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.543804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.543745    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.582787 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.582425    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:48.603631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.603419    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:48.605855 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.605788    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970b169f-9ebf-4fed-b9d1-44ee2f761c58-kube-api-access-kmgf7\" (UniqueName: \"kubernetes.io/projected/970b169f-9ebf-4fed-b9d1-44ee2f761c58-kube-api-access-kmgf7\") pod \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") "
Nov 03 19:45:48.617531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.617148    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-openshift-ca\") pod \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") "
Nov 03 19:45:48.617952 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.617911    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-xks-ca\") pod \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\" (UID: \"970b169f-9ebf-4fed-b9d1-44ee2f761c58\") "
Nov 03 19:45:48.629504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.621444    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-openshift-ca" pod "970b169f-9ebf-4fed-b9d1-44ee2f761c58" (UID: "970b169f-9ebf-4fed-b9d1-44ee2f761c58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:48.629504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.621966    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:48.634265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.634223    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970b169f-9ebf-4fed-b9d1-44ee2f761c58-kube-api-access-kmgf7" pod "970b169f-9ebf-4fed-b9d1-44ee2f761c58" (UID: "970b169f-9ebf-4fed-b9d1-44ee2f761c58"). InnerVolumeSpecName "kube-api-access-kmgf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:48.643583 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.643535    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-xks-ca" pod "970b169f-9ebf-4fed-b9d1-44ee2f761c58" (UID: "970b169f-9ebf-4fed-b9d1-44ee2f761c58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:48.693340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.693105    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:48.725652 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.725563    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970b169f-9ebf-4fed-b9d1-44ee2f761c58-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:48.725652 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.725600    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmgf7\" (UniqueName: \"kubernetes.io/projected/970b169f-9ebf-4fed-b9d1-44ee2f761c58-kube-api-access-kmgf7\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:48.930788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.930755    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"]
Nov 03 19:45:48.934718 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:48.934687    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645da40a_5b03_4200_adba_b2da577b27f3.slice/crio-5f91e934083845a115bab67e95d4ed82822aea85d75cabb1d5cbf3e74b86b6d3 WatchSource:0}: Error finding container 5f91e934083845a115bab67e95d4ed82822aea85d75cabb1d5cbf3e74b86b6d3: Status 404 returned error can't find the container with id 5f91e934083845a115bab67e95d4ed82822aea85d75cabb1d5cbf3e74b86b6d3
Nov 03 19:45:48.939111 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.939054    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.939238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.939189    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.939301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:48.939241    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:49.020722 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:45:49.020692    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5db9016_7dd6_4881_8e89_2bb9751a23c1.slice/crio-0b3d1f5a51226cc8d2f40d362eb6730a90098482518e39d7f213a0bf4ca27703 WatchSource:0}: Error finding container 0b3d1f5a51226cc8d2f40d362eb6730a90098482518e39d7f213a0bf4ca27703: Status 404 returned error can't find the container with id 0b3d1f5a51226cc8d2f40d362eb6730a90098482518e39d7f213a0bf4ca27703
Nov 03 19:45:49.022680 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.022647    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"]
Nov 03 19:45:49.024738 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.024701    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:49.024844 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.024784    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:49.024844 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.024825    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:49.405195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.403582    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" event={"ID":"c6e49e9f-167a-441d-824b-d8499196e409","Type":"ContainerStarted","Data":"8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04"}
Nov 03 19:45:49.434355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.434285    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podStartSLOduration=2.434263352 podStartE2EDuration="2.434263352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.430708482 +0000 UTC m=+862.700831778" watchObservedRunningTime="2026-11-03 19:45:49.434263352 +0000 UTC m=+862.704386641"
Nov 03 19:45:49.511096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.511057    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" event={"ID":"d5db9016-7dd6-4881-8e89-2bb9751a23c1","Type":"ContainerStarted","Data":"12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305"}
Nov 03 19:45:49.511357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.511334    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" event={"ID":"d5db9016-7dd6-4881-8e89-2bb9751a23c1","Type":"ContainerStarted","Data":"0b3d1f5a51226cc8d2f40d362eb6730a90098482518e39d7f213a0bf4ca27703"}
Nov 03 19:45:49.582194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.582014    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:49.623196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.591145    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" event={"ID":"645da40a-5b03-4200-adba-b2da577b27f3","Type":"ContainerStarted","Data":"492e546580bad772080d5d7e0feb0225f8bdd020d4d014d7fe163122f129dcb1"}
Nov 03 19:45:49.623196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.591238    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" event={"ID":"645da40a-5b03-4200-adba-b2da577b27f3","Type":"ContainerStarted","Data":"5f91e934083845a115bab67e95d4ed82822aea85d75cabb1d5cbf3e74b86b6d3"}
Nov 03 19:45:49.642199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.625293    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:45:49.668587 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.668489    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" event={"ID":"23301cd8-4fc8-4544-9744-4fa7cf53ff7c","Type":"ContainerStarted","Data":"9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3"}
Nov 03 19:45:49.668849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.668827    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" event={"ID":"23301cd8-4fc8-4544-9744-4fa7cf53ff7c","Type":"ContainerStarted","Data":"2af372d5ebd04110269fb747f24da50a7d666f76b3a2ea73d125b3ffb2942ed2"}
Nov 03 19:45:49.715199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.695198    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podStartSLOduration=1.695161003 podStartE2EDuration="1.695161003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.582339178 +0000 UTC m=+862.852462474" watchObservedRunningTime="2026-11-03 19:45:49.695161003 +0000 UTC m=+862.965284299"
Nov 03 19:45:49.777198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.696139    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podStartSLOduration=2.6961295180000002 podStartE2EDuration="2.696129518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.651207834 +0000 UTC m=+862.921331127" watchObservedRunningTime="2026-11-03 19:45:49.696129518 +0000 UTC m=+862.966252819"
Nov 03 19:45:49.781249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.697272    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:45:49.781556 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.781526    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv" event={"ID":"970b169f-9ebf-4fed-b9d1-44ee2f761c58","Type":"ContainerDied","Data":"a1f4e5866d0728627e0e06808674541bea4896086d00178ead92fb0b22fb0d59"}
Nov 03 19:45:49.781687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.781672    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f4e5866d0728627e0e06808674541bea4896086d00178ead92fb0b22fb0d59"
Nov 03 19:45:49.781785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.781772    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" event={"ID":"dd9883aa-fca8-416e-8667-85354d4c2702","Type":"ContainerStarted","Data":"d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421"}
Nov 03 19:45:49.781894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.781881    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" event={"ID":"dd9883aa-fca8-416e-8667-85354d4c2702","Type":"ContainerStarted","Data":"f2ddc1f6ec24483e9923a96c4dfdf99e60c694ccc8169446a4f1de4e5b63c045"}
Nov 03 19:45:49.781995 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.724652    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"
Nov 03 19:45:49.854500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.854413    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podStartSLOduration=2.854387458 podStartE2EDuration="2.854387458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.690787562 +0000 UTC m=+862.960910858" watchObservedRunningTime="2026-11-03 19:45:49.854387458 +0000 UTC m=+863.124510764"
Nov 03 19:45:49.872194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.857345    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podStartSLOduration=2.857323966 podStartE2EDuration="2.857323966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.801122759 +0000 UTC m=+863.071246055" watchObservedRunningTime="2026-11-03 19:45:49.857323966 +0000 UTC m=+863.127447262"
Nov 03 19:45:49.889230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.889195    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:45:49.895814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.895778    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:49.897561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:49.897269    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:45:50.010892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.010805    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:45:50.024133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.021741    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:50.024457 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.024425    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:45:50.062263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.062230    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:45:50.073164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.071312    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:50.073164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.071379    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:45:50.503434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.503366    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:50.626202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.607292    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:45:50.637712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.637301    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:50.637712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.637372    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:45:50.675573 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:45:50.675494    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:45:50.701703 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.701360    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:50.701703 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.701423    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:45:50.888154 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.887673    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:50.888154 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.887737    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:45:50.957935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:50.957304    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.034229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.018778    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:51.034229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.018849    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:45:51.064712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.062005    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:51.064712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.062065    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:45:51.064712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.064156    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:45:51.084893 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.084856    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:45:51.199376 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.199269    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.235339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.235286    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.236410 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.236381    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.258747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.258707    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.425111 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.425072    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:51.515364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.515237    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:45:51.522321 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.522279    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:45:51.542264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.528716    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.552527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.552484    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.608646 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.608609    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:51.608887 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.608860    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:45:51.609457 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.609428    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.624194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.609938    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:45:51.711581 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.711417    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.712314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.712233    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:51.712473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.712455    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:45:51.972352 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.972306    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.999299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:51.999265    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:45:52.072390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.072358    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.535203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.532691    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:45:52.535203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.089438    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:45:52.535931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.230570    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.537096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.087486    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.540463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.203669    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.540463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.538525    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:45:52.540463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.282265    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.540463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.248205    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.567206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.266339    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:45:52.567206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.555303    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:45:52.567206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.555400    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:45:52.567206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.555445    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:45:52.567206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.555482    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:45:52.567206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.555517    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:45:52.567206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.248244    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:45:52.567206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.532465    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.567206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.532166    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.567206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.566903    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:45:52.567206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.566960    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:45:52.575612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.568305    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.575612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.568656    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:45:52.598977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.598940    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.599355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.599290    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:45:52.629726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.629676    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.630031 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.629984    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:45:52.630274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.630239    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.706239 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.702097    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.706239 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.702156    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:45:52.937200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.923099    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.937531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:52.937489    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:45:53.028582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.021573    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:53.028582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.021630    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:45:53.028582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.024410    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:45:53.160656 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.128304    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:53.160656 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.159550    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:45:53.249341 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.249233    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:45:53.253309 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.253228    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:53.565819 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.565719    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:53.594197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.566982    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:53.595373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.571247    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:45:53.595630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.572226    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:53.595803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.572272    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:45:53.596104 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.596081    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:45:53.636237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.572297    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:45:54.192195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.598030    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.232883 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.599021    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.233319 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.599068    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.233613 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.706056    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.233784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.233758    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:45:54.233916 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.799400    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.294196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.290162    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:45:54.322655 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.720263    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.347236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:53.706139    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.347236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.231284    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.357220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.026422    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.357220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.354540    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:45:54.357220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.034735    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.357220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.354654    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:45:54.357220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.166093    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.357220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.354733    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:45:54.390497 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.322899    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:45:54.587661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.587558    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.628291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.593313    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:45:54.628804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.602530    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:45:54.629156 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.602558    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.629550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.602573    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.655066 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.602588    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.669200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.622193    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.669200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.657513    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:45:54.738197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.734840    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.738197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.734907    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:45:54.738197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.737578    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded"
Nov 03 19:45:55.015199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.994914    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.015199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:54.994984    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:45:55.059329 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.059288    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.059615 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.059582    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:45:55.071570 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.071528    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:55.072677 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.072645    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.072879 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.072857    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:45:55.188614 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.188565    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:55.236021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.235895    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:55.236606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.235753    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:55.237161 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.235960    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:55.263488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.263434    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:45:55.335370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.323942    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:45:55.335370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.324448    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:45:55.363226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.340743    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:55.653208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.643530    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:55.653208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.644490    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.653208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.644529    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:45:55.653208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.644593    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:55.653208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.644823    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:45:55.705215 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.667776    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:45:55.705215 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.668118    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:45:55.847530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.847497    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.847861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.847812    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:45:55.907260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.881929    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Nov 03 19:45:55.907580 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.887129    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.907750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:55.907721    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:45:56.021066 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.021018    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:56.021283 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.021090    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:45:56.104855 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.104810    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:56.105165 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.105129    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:45:56.189395 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.189275    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.247197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.241333    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:45:56.247197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.241543    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.247197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.242309    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.268562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.267727    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.325388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.325325    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.343373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.343320    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.448795 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.448673    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.601635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.601601    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:45:56.611874 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.611837    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:56.612164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.612134    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:45:56.689811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.689305    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.741313 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.724360    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:56.741313 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.724424    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:45:56.741313 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.724485    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:45:56.741313 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.724786    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:45:56.747073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.741777    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:45:56.747073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.742223    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:45:56.769196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.753310    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:56.769196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.753379    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.914222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.906497    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:56.914222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:56.906558    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:45:57.079259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.076743    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:57.079259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.076811    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:45:57.079259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.078543    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:57.079259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.078590    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:45:57.202656 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.202611    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.241913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.241853    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.243500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.243464    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.275206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.271954    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.346281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.343908    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.346644 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.346596    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:45:57.609699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.609601    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:57.609970 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.609933    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:45:57.696199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.693285    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.701209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.698539    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:57.701209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.698593    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:45:57.725202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.725131    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.725933 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.725476    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:45:57.746600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.746542    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.748459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.748424    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.954027 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:57.953985    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.052637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.044269    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:45:58.052637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.029568    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.052637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.044787    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:45:58.062985 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.062947    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.132357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.132306    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:45:58.272599 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.272473    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.313225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.313029    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.318561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.313727    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:45:58.322233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.320258    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.347244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.347201    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:45:58.348904 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.348869    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.349142 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.349119    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.608724 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.608625    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.617315 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.617254    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:45:58.753197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.751350    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:45:58.753197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.753079    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"]
Nov 03 19:45:58.765199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.754769    4531 cpu_manager.go:414] "containerMap: removing container" podUID="970b169f-9ebf-4fed-b9d1-44ee2f761c58" containerName="revoke-keys"
Nov 03 19:45:58.765199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.754802    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="970b169f-9ebf-4fed-b9d1-44ee2f761c58" containerName="revoke-keys"
Nov 03 19:45:58.789193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.787395    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.789193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.788086    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.797211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.795463    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.797211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.796310    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:45:58.826210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.796473    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.826210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.810545    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:45:58.826210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.810497    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:45:58.826210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.810656    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.857598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.857555    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="970b169f-9ebf-4fed-b9d1-44ee2f761c58" containerName="revoke-keys"
Nov 03 19:45:58.890847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.890797    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.891163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:58.891134    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:45:59.028196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.027366    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:45:59.111661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.060511    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:59.111661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.060576    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:45:59.299254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.298566    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"]
Nov 03 19:45:59.454496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.354596    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"]
Nov 03 19:45:59.454496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.362787    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.454496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.378545    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.454496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.379717    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.454496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.386375    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.454496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.410959    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.454496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.421274    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.454496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.443135    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:59.454496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.443205    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:45:59.454496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.443693    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.454496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.445267    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.482204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.468021    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:45:59.482204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.468140    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:45:59.518871 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.484377    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e116f95-wpnwg\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:45:59.518871 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.484614    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e116f95-wpnwg\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:45:59.518871 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.484893    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmh8t\" (UniqueName: \"kubernetes.io/projected/5fd16332-cc45-4642-8761-d5481a9a7453-kube-api-access-zmh8t\") pod \"maas-api-revoke-sub-2e116f95-wpnwg\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:45:59.564247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.537771    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:59.576906 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.568153    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:45:59.589741 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.589358    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"]
Nov 03 19:45:59.628270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.590610    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e116f95-wpnwg\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:45:59.628270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.590770    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e116f95-wpnwg\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:45:59.628270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.590967    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmh8t\" (UniqueName: \"kubernetes.io/projected/5fd16332-cc45-4642-8761-d5481a9a7453-kube-api-access-zmh8t\") pod \"maas-api-revoke-sub-2e116f95-wpnwg\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:45:59.628270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.595198    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e116f95-wpnwg\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:45:59.628270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.626952    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmh8t\" (UniqueName: \"kubernetes.io/projected/5fd16332-cc45-4642-8761-d5481a9a7453-kube-api-access-zmh8t\") pod \"maas-api-revoke-sub-2e116f95-wpnwg\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:45:59.653051 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.629057    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:59.653051 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.629115    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:45:59.736386 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.710719    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:59.736386 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.710794    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:45:59.740920 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.739876    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e116f95-wpnwg\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:45:59.753141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.752323    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.801750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.801703    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.837279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.810344    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xxvb\" (UniqueName: \"kubernetes.io/projected/b5e1961a-46b9-4994-9887-80b5236bfbe2-kube-api-access-8xxvb\") pod \"maas-api-revoke-sub-1e3e60c3-qw4f6\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:45:59.837879 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.837830    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e3e60c3-qw4f6\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:45:59.838264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.838229    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e3e60c3-qw4f6\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:45:59.838647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.838607    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.844381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.840787    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.844381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.842137    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.891196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.887444    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:59.916325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.916276    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:45:59.942197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.941308    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xxvb\" (UniqueName: \"kubernetes.io/projected/b5e1961a-46b9-4994-9887-80b5236bfbe2-kube-api-access-8xxvb\") pod \"maas-api-revoke-sub-1e3e60c3-qw4f6\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:45:59.942197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.941506    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e3e60c3-qw4f6\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:45:59.942197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.941626    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e3e60c3-qw4f6\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:45:59.943603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.943578    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:45:59.943830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.943804    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e3e60c3-qw4f6\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:45:59.964228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:45:59.959670    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e3e60c3-qw4f6\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:46:00.012666 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.012636    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:46:00.017336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.017306    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:46:00.017551 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.017526    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:46:00.073867 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.073829    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:46:00.074293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.074259    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:46:00.215897 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.215860    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xxvb\" (UniqueName: \"kubernetes.io/projected/b5e1961a-46b9-4994-9887-80b5236bfbe2-kube-api-access-8xxvb\") pod \"maas-api-revoke-sub-1e3e60c3-qw4f6\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:46:00.322456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.322420    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:46:00.407732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.406322    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.407732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.406766    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:00.407732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.407078    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:46:00.486235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.469891    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:46:00.530272 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.527525    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.608827 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.608785    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:46:00.609084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.609051    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:46:00.733250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.725871    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:46:00.733250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.725936    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:46:00.800025 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.799980    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:46:00.803294 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.803244    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.852546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.852505    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.854283 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.854244    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:46:00.855282 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.854448    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.889617 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.889581    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:46:00.906387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:00.906340    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 19:46:01.016100 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.016002    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:46:01.019401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.019354    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:46:01.084198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.071818    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:46:01.084198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.071881    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:46:01.289665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.256117    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:46:01.376715 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.358438    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.416207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.412303    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:46:01.416207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.412266    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.441199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.416709    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:01.507201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.476516    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.541251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.527731    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.718482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.669260    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.718482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.686278    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:46:01.718482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.686337    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 19:46:01.718482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.702844    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:46:01.718482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.702895    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:46:01.800912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.800834    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.835271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.803937    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.857346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.857275    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:46:01.857679 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.857572    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.857870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.857806    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:46:01.968427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.965394    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"]
Nov 03 19:46:01.968427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.965595    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:46:01.968427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:01.965665    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:46:02.018280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.018233    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:46:02.018541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.018511    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 19:46:02.442234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.432004    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:46:02.442234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.432081    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 19:46:02.486340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.486306    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:46:02.517208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.509061    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:46:02.517208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.511923    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:02.834692 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.834661    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:46:02.834941 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.834910    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 19:46:02.837014 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.836986    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:46:02.852987 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.835164    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:02.869080 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.865813    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:02.874193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.872525    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:02.927089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.927059    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"]
Nov 03 19:46:02.945903 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.945853    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:02.946152 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:02.946126    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:03.081935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.081904    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:46:03.082432 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.082391    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6" event={"ID":"b5e1961a-46b9-4994-9887-80b5236bfbe2","Type":"ContainerStarted","Data":"24c80717d4bfb36e910ca179349cb8e76cfef88edc9bf90e001c6949bd5bc524"}
Nov 03 19:46:03.082687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.082675    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:46:03.082817 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.082804    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg" event={"ID":"5fd16332-cc45-4642-8761-d5481a9a7453","Type":"ContainerStarted","Data":"9b9d217b77efa32937ba67724167be3affceb703baeb5b7ad477a4b1664abe51"}
Nov 03 19:46:03.082949 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.082939    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:46:03.083049 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.083041    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:46:03.083158 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.083149    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:46:03.083284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.083275    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:46:03.083363 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.083356    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:46:03.083479 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.083470    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:46:03.083560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.083552    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:46:03.094266 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.094246    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 19:46:03.123405 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.122486    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:46:03.409856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.409829    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:46:03.486443 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.486411    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:46:03.551240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.551143    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:46:03.623654 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.619469    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:46:03.693913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.693885    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:46:03.719834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.719805    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:46:03.849867 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:03.849771    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:46:04.008944 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:04.008153    4531 generic.go:356] "Generic (PLEG): container finished" podID="b5e1961a-46b9-4994-9887-80b5236bfbe2" containerID="02de99a85d61c171cd9ce4c9b1af8c0b7d870b0b20523fd2033ceca8d26a2a2f" exitCode=0
Nov 03 19:46:04.009080 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:04.008289    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6" event={"ID":"b5e1961a-46b9-4994-9887-80b5236bfbe2","Type":"ContainerDied","Data":"02de99a85d61c171cd9ce4c9b1af8c0b7d870b0b20523fd2033ceca8d26a2a2f"}
Nov 03 19:46:04.011766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:04.011739    4531 generic.go:356] "Generic (PLEG): container finished" podID="5fd16332-cc45-4642-8761-d5481a9a7453" containerID="3883c45641929d03ef302a74242192571cdf683fc2654cf78db7a29f3d378251" exitCode=0
Nov 03 19:46:04.011872 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:04.011847    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg" event={"ID":"5fd16332-cc45-4642-8761-d5481a9a7453","Type":"ContainerDied","Data":"3883c45641929d03ef302a74242192571cdf683fc2654cf78db7a29f3d378251"}
Nov 03 19:46:04.013087 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:04.013067    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:46:04.013209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:04.013135    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 19:46:04.062244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:04.062159    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:04.062678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:04.062629    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:04.893639 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:04.893599    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:46:04.896990 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:04.896963    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 19:46:05.251440 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.251413    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:46:05.259518 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.259495    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:46:05.362227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.353435    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:05.373638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.370581    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-openshift-ca\") pod \"b5e1961a-46b9-4994-9887-80b5236bfbe2\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") "
Nov 03 19:46:05.373638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.370654    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e1961a-46b9-4994-9887-80b5236bfbe2-kube-api-access-8xxvb\" (UniqueName: \"kubernetes.io/projected/b5e1961a-46b9-4994-9887-80b5236bfbe2-kube-api-access-8xxvb\") pod \"b5e1961a-46b9-4994-9887-80b5236bfbe2\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") "
Nov 03 19:46:05.373638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.370698    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd16332-cc45-4642-8761-d5481a9a7453-kube-api-access-zmh8t\" (UniqueName: \"kubernetes.io/projected/5fd16332-cc45-4642-8761-d5481a9a7453-kube-api-access-zmh8t\") pod \"5fd16332-cc45-4642-8761-d5481a9a7453\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") "
Nov 03 19:46:05.373638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.372052    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-openshift-ca\") pod \"5fd16332-cc45-4642-8761-d5481a9a7453\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") "
Nov 03 19:46:05.373638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.372148    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-xks-ca\") pod \"b5e1961a-46b9-4994-9887-80b5236bfbe2\" (UID: \"b5e1961a-46b9-4994-9887-80b5236bfbe2\") "
Nov 03 19:46:05.373638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.372356    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-xks-ca\") pod \"5fd16332-cc45-4642-8761-d5481a9a7453\" (UID: \"5fd16332-cc45-4642-8761-d5481a9a7453\") "
Nov 03 19:46:05.393434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.386272    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-openshift-ca" pod "5fd16332-cc45-4642-8761-d5481a9a7453" (UID: "5fd16332-cc45-4642-8761-d5481a9a7453"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:05.393434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.386657    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-openshift-ca" pod "b5e1961a-46b9-4994-9887-80b5236bfbe2" (UID: "b5e1961a-46b9-4994-9887-80b5236bfbe2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:05.393434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.393316    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-xks-ca" pod "5fd16332-cc45-4642-8761-d5481a9a7453" (UID: "5fd16332-cc45-4642-8761-d5481a9a7453"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:05.399261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.394970    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd16332-cc45-4642-8761-d5481a9a7453-kube-api-access-zmh8t" pod "5fd16332-cc45-4642-8761-d5481a9a7453" (UID: "5fd16332-cc45-4642-8761-d5481a9a7453"). InnerVolumeSpecName "kube-api-access-zmh8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:05.399261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.395061    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e1961a-46b9-4994-9887-80b5236bfbe2-kube-api-access-8xxvb" pod "b5e1961a-46b9-4994-9887-80b5236bfbe2" (UID: "b5e1961a-46b9-4994-9887-80b5236bfbe2"). InnerVolumeSpecName "kube-api-access-8xxvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:05.399261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.395435    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-xks-ca" pod "b5e1961a-46b9-4994-9887-80b5236bfbe2" (UID: "b5e1961a-46b9-4994-9887-80b5236bfbe2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:05.474136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.474029    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:05.474136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.474083    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:05.474136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.474094    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5e1961a-46b9-4994-9887-80b5236bfbe2-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:05.474136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.474103    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xxvb\" (UniqueName: \"kubernetes.io/projected/b5e1961a-46b9-4994-9887-80b5236bfbe2-kube-api-access-8xxvb\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:05.474136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.474114    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmh8t\" (UniqueName: \"kubernetes.io/projected/5fd16332-cc45-4642-8761-d5481a9a7453-kube-api-access-zmh8t\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:05.474136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:05.474123    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd16332-cc45-4642-8761-d5481a9a7453-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:06.030268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:06.030232    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"
Nov 03 19:46:06.030924 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:06.030894    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6" event={"ID":"b5e1961a-46b9-4994-9887-80b5236bfbe2","Type":"ContainerDied","Data":"24c80717d4bfb36e910ca179349cb8e76cfef88edc9bf90e001c6949bd5bc524"}
Nov 03 19:46:06.031102 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:06.030944    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24c80717d4bfb36e910ca179349cb8e76cfef88edc9bf90e001c6949bd5bc524"
Nov 03 19:46:06.033475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:06.033448    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg" event={"ID":"5fd16332-cc45-4642-8761-d5481a9a7453","Type":"ContainerDied","Data":"9b9d217b77efa32937ba67724167be3affceb703baeb5b7ad477a4b1664abe51"}
Nov 03 19:46:06.033594 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:06.033501    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b9d217b77efa32937ba67724167be3affceb703baeb5b7ad477a4b1664abe51"
Nov 03 19:46:06.033594 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:06.033565    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"
Nov 03 19:46:07.390194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:07.388099    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:09.259665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:09.259633    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:10.836824 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.836783    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"]
Nov 03 19:46:10.837548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.837531    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5fd16332-cc45-4642-8761-d5481a9a7453" containerName="revoke-keys"
Nov 03 19:46:10.837603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.837549    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd16332-cc45-4642-8761-d5481a9a7453" containerName="revoke-keys"
Nov 03 19:46:10.837603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.837568    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b5e1961a-46b9-4994-9887-80b5236bfbe2" containerName="revoke-keys"
Nov 03 19:46:10.837603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.837574    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e1961a-46b9-4994-9887-80b5236bfbe2" containerName="revoke-keys"
Nov 03 19:46:10.837701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.837688    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd16332-cc45-4642-8761-d5481a9a7453" containerName="revoke-keys"
Nov 03 19:46:10.837701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.837700    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e1961a-46b9-4994-9887-80b5236bfbe2" containerName="revoke-keys"
Nov 03 19:46:10.843057 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.843031    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:10.845791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.845764    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:46:10.846347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.846325    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"]
Nov 03 19:46:10.848246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.848224    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:46:10.876604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.876570    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkl2p\" (UniqueName: \"kubernetes.io/projected/74aaaa79-143d-4be2-a06d-c304b6c34b5f-kube-api-access-tkl2p\") pod \"maas-api-revoke-sub-9831930f-v2ths\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:10.876766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.876634    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9831930f-v2ths\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:10.876766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.876752    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9831930f-v2ths\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:10.938099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.938056    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"]
Nov 03 19:46:10.942814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.942786    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:10.950707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.950675    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"]
Nov 03 19:46:10.978649 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.978618    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkl2p\" (UniqueName: \"kubernetes.io/projected/74aaaa79-143d-4be2-a06d-c304b6c34b5f-kube-api-access-tkl2p\") pod \"maas-api-revoke-sub-9831930f-v2ths\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:10.978845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.978696    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9831930f-v2ths\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:10.978845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.978731    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwng\" (UniqueName: \"kubernetes.io/projected/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-kube-api-access-hcwng\") pod \"maas-api-revoke-sub-590f43e5-m2zhf\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:10.978977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.978887    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590f43e5-m2zhf\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:10.978977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.978914    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9831930f-v2ths\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:10.979092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.979048    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590f43e5-m2zhf\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:10.979726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.979703    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9831930f-v2ths\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:10.981318 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.981296    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9831930f-v2ths\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:10.985934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:10.985913    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkl2p\" (UniqueName: \"kubernetes.io/projected/74aaaa79-143d-4be2-a06d-c304b6c34b5f-kube-api-access-tkl2p\") pod \"maas-api-revoke-sub-9831930f-v2ths\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:11.074119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.074083    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:11.080782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.080751    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590f43e5-m2zhf\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:11.080931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.080901    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590f43e5-m2zhf\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:11.081128 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.081111    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwng\" (UniqueName: \"kubernetes.io/projected/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-kube-api-access-hcwng\") pod \"maas-api-revoke-sub-590f43e5-m2zhf\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:11.081634 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.081612    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590f43e5-m2zhf\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:11.083538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.083520    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590f43e5-m2zhf\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:11.088231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.088144    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwng\" (UniqueName: \"kubernetes.io/projected/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-kube-api-access-hcwng\") pod \"maas-api-revoke-sub-590f43e5-m2zhf\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:11.161404 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.161368    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:11.263479 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.263324    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:11.470852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.470768    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:11.480253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.476561    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"]
Nov 03 19:46:11.554681 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:11.554653    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8fa3c8_7ff0_4a06_9ddf_54010de2c1ff.slice/crio-2a23be8d11d368a3635f374821bac28e47919035ec29c3ba45873664b3615567 WatchSource:0}: Error finding container 2a23be8d11d368a3635f374821bac28e47919035ec29c3ba45873664b3615567: Status 404 returned error can't find the container with id 2a23be8d11d368a3635f374821bac28e47919035ec29c3ba45873664b3615567
Nov 03 19:46:11.555965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:11.555934    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"]
Nov 03 19:46:12.082329 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.082288    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf" event={"ID":"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff","Type":"ContainerStarted","Data":"98d7af69bc6817a24d3164ed4a543def2eb5ab9ec269c453552be626ab7f5d44"}
Nov 03 19:46:12.083295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.082417    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf" event={"ID":"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff","Type":"ContainerStarted","Data":"2a23be8d11d368a3635f374821bac28e47919035ec29c3ba45873664b3615567"}
Nov 03 19:46:12.084224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.084199    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths" event={"ID":"74aaaa79-143d-4be2-a06d-c304b6c34b5f","Type":"ContainerStarted","Data":"4030faad0f0d8af23a6de3daf65cbe39942cf9e0784116843b91e2d2fe634f6e"}
Nov 03 19:46:12.084318 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.084244    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths" event={"ID":"74aaaa79-143d-4be2-a06d-c304b6c34b5f","Type":"ContainerStarted","Data":"51fd149256ca035bd29b02bb247e548b32d3b41ca4c78817b6741cd1948ae094"}
Nov 03 19:46:12.100350 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.099939    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf" podStartSLOduration=2.099921109 podStartE2EDuration="2.099921109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:12.096383936 +0000 UTC m=+885.366507231" watchObservedRunningTime="2026-11-03 19:46:12.099921109 +0000 UTC m=+885.370044406"
Nov 03 19:46:12.118103 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.118048    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths" podStartSLOduration=2.118034273 podStartE2EDuration="2.118034273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:12.112512905 +0000 UTC m=+885.382636201" watchObservedRunningTime="2026-11-03 19:46:12.118034273 +0000 UTC m=+885.388157568"
Nov 03 19:46:12.194788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.194754    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"]
Nov 03 19:46:12.199522 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.199483    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:12.207487 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.207455    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"]
Nov 03 19:46:12.300199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.296312    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:12.340206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.334660    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-956950be-zffj7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:12.356473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.356382    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-956950be-zffj7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:12.357547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.357029    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l264p\" (UniqueName: \"kubernetes.io/projected/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-kube-api-access-l264p\") pod \"maas-api-revoke-sub-956950be-zffj7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:12.458922 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.458892    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-956950be-zffj7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:12.459088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.458985    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-956950be-zffj7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:12.459326 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.459299    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l264p\" (UniqueName: \"kubernetes.io/projected/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-kube-api-access-l264p\") pod \"maas-api-revoke-sub-956950be-zffj7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:12.459675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.459654    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-956950be-zffj7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:12.461207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.461186    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-956950be-zffj7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:12.466255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.466230    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l264p\" (UniqueName: \"kubernetes.io/projected/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-kube-api-access-l264p\") pod \"maas-api-revoke-sub-956950be-zffj7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:12.523913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.523887    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:12.807626 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:12.807587    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"]
Nov 03 19:46:12.808158 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:12.808128    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432d8dcf_a8c0_4130_8e8e_d86d219b78f7.slice/crio-d215e7ed82eadd3e9d0ee4e1034aa18b8412cb40b108ef3d25dd073a3430cc68 WatchSource:0}: Error finding container d215e7ed82eadd3e9d0ee4e1034aa18b8412cb40b108ef3d25dd073a3430cc68: Status 404 returned error can't find the container with id d215e7ed82eadd3e9d0ee4e1034aa18b8412cb40b108ef3d25dd073a3430cc68
Nov 03 19:46:13.093759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:13.093730    4531 generic.go:356] "Generic (PLEG): container finished" podID="fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff" containerID="98d7af69bc6817a24d3164ed4a543def2eb5ab9ec269c453552be626ab7f5d44" exitCode=0
Nov 03 19:46:13.094217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:13.093814    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf" event={"ID":"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff","Type":"ContainerDied","Data":"98d7af69bc6817a24d3164ed4a543def2eb5ab9ec269c453552be626ab7f5d44"}
Nov 03 19:46:13.095535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:13.095506    4531 generic.go:356] "Generic (PLEG): container finished" podID="74aaaa79-143d-4be2-a06d-c304b6c34b5f" containerID="4030faad0f0d8af23a6de3daf65cbe39942cf9e0784116843b91e2d2fe634f6e" exitCode=0
Nov 03 19:46:13.095647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:13.095584    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths" event={"ID":"74aaaa79-143d-4be2-a06d-c304b6c34b5f","Type":"ContainerDied","Data":"4030faad0f0d8af23a6de3daf65cbe39942cf9e0784116843b91e2d2fe634f6e"}
Nov 03 19:46:13.097014 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:13.096992    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7" event={"ID":"432d8dcf-a8c0-4130-8e8e-d86d219b78f7","Type":"ContainerStarted","Data":"d215e7ed82eadd3e9d0ee4e1034aa18b8412cb40b108ef3d25dd073a3430cc68"}
Nov 03 19:46:13.364250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:13.352539    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:14.030385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.030303    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:14.111151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.111052    4531 generic.go:356] "Generic (PLEG): container finished" podID="432d8dcf-a8c0-4130-8e8e-d86d219b78f7" containerID="b1af3cfa42e3ae01a30b206e671cb4201f4fc9041c50d83661350b5a149ce0e0" exitCode=0
Nov 03 19:46:14.112099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.112060    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7" event={"ID":"432d8dcf-a8c0-4130-8e8e-d86d219b78f7","Type":"ContainerDied","Data":"b1af3cfa42e3ae01a30b206e671cb4201f4fc9041c50d83661350b5a149ce0e0"}
Nov 03 19:46:14.329980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.329952    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:14.334164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.334139    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:14.392049 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.392015    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-openshift-ca\") pod \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") "
Nov 03 19:46:14.392270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.392201    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-xks-ca\") pod \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") "
Nov 03 19:46:14.392349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.392298    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-openshift-ca\") pod \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") "
Nov 03 19:46:14.392447 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.392429    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-kube-api-access-hcwng\" (UniqueName: \"kubernetes.io/projected/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-kube-api-access-hcwng\") pod \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") "
Nov 03 19:46:14.392547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.392531    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-xks-ca\") pod \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\" (UID: \"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff\") "
Nov 03 19:46:14.392627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.392610    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74aaaa79-143d-4be2-a06d-c304b6c34b5f-kube-api-access-tkl2p\" (UniqueName: \"kubernetes.io/projected/74aaaa79-143d-4be2-a06d-c304b6c34b5f-kube-api-access-tkl2p\") pod \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\" (UID: \"74aaaa79-143d-4be2-a06d-c304b6c34b5f\") "
Nov 03 19:46:14.392734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.392428    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-openshift-ca" pod "fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff" (UID: "fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:14.392799 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.392704    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-openshift-ca" pod "74aaaa79-143d-4be2-a06d-c304b6c34b5f" (UID: "74aaaa79-143d-4be2-a06d-c304b6c34b5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:14.393598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.393570    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:14.393739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.393603    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:14.395370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.395159    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-kube-api-access-hcwng" pod "fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff" (UID: "fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff"). InnerVolumeSpecName "kube-api-access-hcwng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:14.395370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.395335    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74aaaa79-143d-4be2-a06d-c304b6c34b5f-kube-api-access-tkl2p" pod "74aaaa79-143d-4be2-a06d-c304b6c34b5f" (UID: "74aaaa79-143d-4be2-a06d-c304b6c34b5f"). InnerVolumeSpecName "kube-api-access-tkl2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:14.395540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.395480    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-xks-ca" pod "fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff" (UID: "fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:14.396074 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.396044    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-xks-ca" pod "74aaaa79-143d-4be2-a06d-c304b6c34b5f" (UID: "74aaaa79-143d-4be2-a06d-c304b6c34b5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:14.495460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.495422    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkl2p\" (UniqueName: \"kubernetes.io/projected/74aaaa79-143d-4be2-a06d-c304b6c34b5f-kube-api-access-tkl2p\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:14.495460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.495463    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74aaaa79-143d-4be2-a06d-c304b6c34b5f-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:14.495653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.495480    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcwng\" (UniqueName: \"kubernetes.io/projected/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-kube-api-access-hcwng\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:14.495653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.495494    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:14.697201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.697087    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"]
Nov 03 19:46:14.697802 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.697786    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff" containerName="revoke-keys"
Nov 03 19:46:14.697858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.697803    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff" containerName="revoke-keys"
Nov 03 19:46:14.697858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.697843    4531 cpu_manager.go:414] "containerMap: removing container" podUID="74aaaa79-143d-4be2-a06d-c304b6c34b5f" containerName="revoke-keys"
Nov 03 19:46:14.697858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.697848    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74aaaa79-143d-4be2-a06d-c304b6c34b5f" containerName="revoke-keys"
Nov 03 19:46:14.697988 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.697952    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="74aaaa79-143d-4be2-a06d-c304b6c34b5f" containerName="revoke-keys"
Nov 03 19:46:14.697988 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.697963    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff" containerName="revoke-keys"
Nov 03 19:46:14.702026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.701945    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:14.710835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.710805    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"]
Nov 03 19:46:14.800042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.799998    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k582t\" (UniqueName: \"kubernetes.io/projected/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-kube-api-access-k582t\") pod \"maas-api-revoke-sub-758548f8-trktg\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:14.800305 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.800126    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758548f8-trktg\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:14.800305 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.800292    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758548f8-trktg\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:14.882202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.879920    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:14.903773 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.903733    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k582t\" (UniqueName: \"kubernetes.io/projected/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-kube-api-access-k582t\") pod \"maas-api-revoke-sub-758548f8-trktg\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:14.904064 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.904038    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758548f8-trktg\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:14.904349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.904328    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758548f8-trktg\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:14.905367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.905346    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758548f8-trktg\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:14.909037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.909017    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758548f8-trktg\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:14.912056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:14.912032    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k582t\" (UniqueName: \"kubernetes.io/projected/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-kube-api-access-k582t\") pod \"maas-api-revoke-sub-758548f8-trktg\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:15.023133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.023048    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:15.124153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.123752    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"
Nov 03 19:46:15.124153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.123755    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths" event={"ID":"74aaaa79-143d-4be2-a06d-c304b6c34b5f","Type":"ContainerDied","Data":"51fd149256ca035bd29b02bb247e548b32d3b41ca4c78817b6741cd1948ae094"}
Nov 03 19:46:15.124153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.123902    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51fd149256ca035bd29b02bb247e548b32d3b41ca4c78817b6741cd1948ae094"
Nov 03 19:46:15.126245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.126221    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"
Nov 03 19:46:15.126433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.126403    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf" event={"ID":"fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff","Type":"ContainerDied","Data":"2a23be8d11d368a3635f374821bac28e47919035ec29c3ba45873664b3615567"}
Nov 03 19:46:15.126525 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.126462    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a23be8d11d368a3635f374821bac28e47919035ec29c3ba45873664b3615567"
Nov 03 19:46:15.304562 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:15.304522    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c79918_75c1_40f5_bbfe_5fdcf99f76c5.slice/crio-f98ae41767f984e896e47713e9fe1e2bd885051b1a638bca8ecf99fec1e1138b WatchSource:0}: Error finding container f98ae41767f984e896e47713e9fe1e2bd885051b1a638bca8ecf99fec1e1138b: Status 404 returned error can't find the container with id f98ae41767f984e896e47713e9fe1e2bd885051b1a638bca8ecf99fec1e1138b
Nov 03 19:46:15.305285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.305260    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"]
Nov 03 19:46:15.331455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.331424    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:15.433138 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.433113    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:15.513993 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.513952    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-kube-api-access-l264p\" (UniqueName: \"kubernetes.io/projected/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-kube-api-access-l264p\") pod \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") "
Nov 03 19:46:15.514271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.514080    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-xks-ca\") pod \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") "
Nov 03 19:46:15.514357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.514333    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-openshift-ca\") pod \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\" (UID: \"432d8dcf-a8c0-4130-8e8e-d86d219b78f7\") "
Nov 03 19:46:15.514774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.514749    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-openshift-ca" pod "432d8dcf-a8c0-4130-8e8e-d86d219b78f7" (UID: "432d8dcf-a8c0-4130-8e8e-d86d219b78f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:15.515062 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.515048    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:15.516196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.516155    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-kube-api-access-l264p" pod "432d8dcf-a8c0-4130-8e8e-d86d219b78f7" (UID: "432d8dcf-a8c0-4130-8e8e-d86d219b78f7"). InnerVolumeSpecName "kube-api-access-l264p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:15.516264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.516205    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-xks-ca" pod "432d8dcf-a8c0-4130-8e8e-d86d219b78f7" (UID: "432d8dcf-a8c0-4130-8e8e-d86d219b78f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:15.538520 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.538470    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:15.625900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.621829    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l264p\" (UniqueName: \"kubernetes.io/projected/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-kube-api-access-l264p\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:15.625900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:15.621869    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432d8dcf-a8c0-4130-8e8e-d86d219b78f7-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:16.135307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:16.135280    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"
Nov 03 19:46:16.135307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:16.135295    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7" event={"ID":"432d8dcf-a8c0-4130-8e8e-d86d219b78f7","Type":"ContainerDied","Data":"d215e7ed82eadd3e9d0ee4e1034aa18b8412cb40b108ef3d25dd073a3430cc68"}
Nov 03 19:46:16.135774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:16.135350    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d215e7ed82eadd3e9d0ee4e1034aa18b8412cb40b108ef3d25dd073a3430cc68"
Nov 03 19:46:16.137140 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:16.137114    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg" event={"ID":"24c79918-75c1-40f5-bbfe-5fdcf99f76c5","Type":"ContainerStarted","Data":"0df56b2669aa99c3580402e4d9e07ba4c81c8448e39ac77a8f2b7515f2b66b79"}
Nov 03 19:46:16.137296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:16.137154    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg" event={"ID":"24c79918-75c1-40f5-bbfe-5fdcf99f76c5","Type":"ContainerStarted","Data":"f98ae41767f984e896e47713e9fe1e2bd885051b1a638bca8ecf99fec1e1138b"}
Nov 03 19:46:16.153194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:16.153114    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg" podStartSLOduration=2.153097346 podStartE2EDuration="2.153097346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:16.151665398 +0000 UTC m=+889.421788693" watchObservedRunningTime="2026-11-03 19:46:16.153097346 +0000 UTC m=+889.423220645"
Nov 03 19:46:16.462284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:16.458221    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:17.131196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.128667    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:17.194197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.190097    4531 generic.go:356] "Generic (PLEG): container finished" podID="24c79918-75c1-40f5-bbfe-5fdcf99f76c5" containerID="0df56b2669aa99c3580402e4d9e07ba4c81c8448e39ac77a8f2b7515f2b66b79" exitCode=0
Nov 03 19:46:17.194197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.190613    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg" event={"ID":"24c79918-75c1-40f5-bbfe-5fdcf99f76c5","Type":"ContainerDied","Data":"0df56b2669aa99c3580402e4d9e07ba4c81c8448e39ac77a8f2b7515f2b66b79"}
Nov 03 19:46:17.636664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.636625    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"]
Nov 03 19:46:17.637543 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.637524    4531 cpu_manager.go:414] "containerMap: removing container" podUID="432d8dcf-a8c0-4130-8e8e-d86d219b78f7" containerName="revoke-keys"
Nov 03 19:46:17.637606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.637545    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432d8dcf-a8c0-4130-8e8e-d86d219b78f7" containerName="revoke-keys"
Nov 03 19:46:17.637666 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.637657    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="432d8dcf-a8c0-4130-8e8e-d86d219b78f7" containerName="revoke-keys"
Nov 03 19:46:17.641490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.641468    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.650056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.650008    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"]
Nov 03 19:46:17.796118 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.796080    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.796324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.796139    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.796324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.796224    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9q2k\" (UniqueName: \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-kube-api-access-s9q2k\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.796412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.796374    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.796459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.796419    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.796609 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.796587    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.796647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.796623    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.796709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.796696    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.796750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.796737    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.898727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.898645    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.898727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.898711    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.898919 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.898827    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.898919 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.898850    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.898919 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.898877    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.899044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.898930    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.899098 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.899074    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.899160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.899127    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.899230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.899156    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9q2k\" (UniqueName: \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-kube-api-access-s9q2k\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.899230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.899196    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.899353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.899312    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.899410 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.899362    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.899618 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.899596    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.899849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.899830    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.901122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.901104    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.901255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.901237    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.905928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.905901    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.906001 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.905977    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9q2k\" (UniqueName: \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-kube-api-access-s9q2k\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:17.970856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:17.970820    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:18.074402 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.068532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:18.189189 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.189081    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"]
Nov 03 19:46:18.205477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.205441    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.218350 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.216456    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"]
Nov 03 19:46:18.228689 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.228656    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"]
Nov 03 19:46:18.237363 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.236795    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.241410 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.240670    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"]
Nov 03 19:46:18.306906 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.306868    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.307024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.306911    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.307099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.307023    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkzbv\" (UniqueName: \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-kube-api-access-gkzbv\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.307099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.307059    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.307216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.307151    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.307265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.307245    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1a97c8-d12c-46eb-a635-e450998128d5-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.307321 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.307308    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.307379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.307327    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.307432 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.307384    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.322125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.322087    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"]
Nov 03 19:46:18.322613 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:18.322580    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc42104f_5b1e_4ae2_9a4e_08fe160a4fa2.slice/crio-53d9a3bc0ed466cc69b481792177381d18a625d3875c93a7de3947dd9faf1d19 WatchSource:0}: Error finding container 53d9a3bc0ed466cc69b481792177381d18a625d3875c93a7de3947dd9faf1d19: Status 404 returned error can't find the container with id 53d9a3bc0ed466cc69b481792177381d18a625d3875c93a7de3947dd9faf1d19
Nov 03 19:46:18.327571 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.327529    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.327770 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.327745    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.327957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.327936    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.405626 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.405601    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:18.408448 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.408422    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.408570 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.408505    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.408570 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.408564    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.408710 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.408663    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.408762 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.408721    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1a97c8-d12c-46eb-a635-e450998128d5-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.408861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.408838    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.408918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.408879    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.408975 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.408941    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.409029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.408976    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.409029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409004    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.409131 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409075    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwnt4\" (UniqueName: \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-kube-api-access-jwnt4\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.409204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409143    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.409268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409250    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.409397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409375    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.409487 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409457    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.409547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409492    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.409596 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409540    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1a97c8-d12c-46eb-a635-e450998128d5-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.409596 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409506    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04530f22-b013-40fe-8390-81a0ea009267-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.409665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409626    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkzbv\" (UniqueName: \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-kube-api-access-gkzbv\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.409665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409653    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.409758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409675    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04530f22-b013-40fe-8390-81a0ea009267-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.409758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409712    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.409971 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.409950    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.411507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.411485    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.411749 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.411732    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.423676 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.423647    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkzbv\" (UniqueName: \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-kube-api-access-gkzbv\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.424112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.424087    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.462042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.461950    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"]
Nov 03 19:46:18.463110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.463077    4531 cpu_manager.go:414] "containerMap: removing container" podUID="24c79918-75c1-40f5-bbfe-5fdcf99f76c5" containerName="revoke-keys"
Nov 03 19:46:18.463110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.463110    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c79918-75c1-40f5-bbfe-5fdcf99f76c5" containerName="revoke-keys"
Nov 03 19:46:18.463360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.463341    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c79918-75c1-40f5-bbfe-5fdcf99f76c5" containerName="revoke-keys"
Nov 03 19:46:18.492511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.492478    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"]
Nov 03 19:46:18.492678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.492667    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:18.510732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.510703    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-xks-ca\") pod \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") "
Nov 03 19:46:18.511483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.511460    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-openshift-ca\") pod \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") "
Nov 03 19:46:18.511682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.511666    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-kube-api-access-k582t\" (UniqueName: \"kubernetes.io/projected/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-kube-api-access-k582t\") pod \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\" (UID: \"24c79918-75c1-40f5-bbfe-5fdcf99f76c5\") "
Nov 03 19:46:18.512088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.512067    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.512289 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.512268    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.512472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.512442    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.512814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.512798    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.512962 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.512947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.513097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.513083    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnt4\" (UniqueName: \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-kube-api-access-jwnt4\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.513332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.513307    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.513507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.513493    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04530f22-b013-40fe-8390-81a0ea009267-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.513687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.513674    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04530f22-b013-40fe-8390-81a0ea009267-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.514819 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.514791    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04530f22-b013-40fe-8390-81a0ea009267-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.515675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.515651    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.515759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.515653    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.515834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.515810    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-openshift-ca" pod "24c79918-75c1-40f5-bbfe-5fdcf99f76c5" (UID: "24c79918-75c1-40f5-bbfe-5fdcf99f76c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:18.515883 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.515818    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.516289 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.516242    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.520018 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.519993    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04530f22-b013-40fe-8390-81a0ea009267-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.520452 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.520427    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.521276 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.521216    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-kube-api-access-k582t" pod "24c79918-75c1-40f5-bbfe-5fdcf99f76c5" (UID: "24c79918-75c1-40f5-bbfe-5fdcf99f76c5"). InnerVolumeSpecName "kube-api-access-k582t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:18.524216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.524187    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnt4\" (UniqueName: \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-kube-api-access-jwnt4\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.530082 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.526662    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.530402 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.530363    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-xks-ca" pod "24c79918-75c1-40f5-bbfe-5fdcf99f76c5" (UID: "24c79918-75c1-40f5-bbfe-5fdcf99f76c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:18.531766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.531579    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:18.574809 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.574769    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:18.623660 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.623620    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcfvz\" (UniqueName: \"kubernetes.io/projected/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-kube-api-access-jcfvz\") pod \"maas-api-revoke-sub-51db6282-jcslt\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:18.623826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.623685    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51db6282-jcslt\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:18.623970 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.623951    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51db6282-jcslt\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:18.624230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.624209    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:18.624331 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.624234    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k582t\" (UniqueName: \"kubernetes.io/projected/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-kube-api-access-k582t\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:18.624331 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.624249    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c79918-75c1-40f5-bbfe-5fdcf99f76c5-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:18.725889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.725802    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcfvz\" (UniqueName: \"kubernetes.io/projected/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-kube-api-access-jcfvz\") pod \"maas-api-revoke-sub-51db6282-jcslt\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:18.725889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.725863    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51db6282-jcslt\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:18.726122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.726101    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51db6282-jcslt\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:18.726889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.726865    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51db6282-jcslt\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:18.728655 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.728630    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51db6282-jcslt\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:18.732798 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.732771    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcfvz\" (UniqueName: \"kubernetes.io/projected/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-kube-api-access-jcfvz\") pod \"maas-api-revoke-sub-51db6282-jcslt\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:18.822029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.821992    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"]
Nov 03 19:46:18.823273 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:18.823248    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1a97c8_d12c_46eb_a635_e450998128d5.slice/crio-53c9daf1470dde811a1820f85ef5e5ac9c5c4a8876d6b7870e2fff51d7b1676d WatchSource:0}: Error finding container 53c9daf1470dde811a1820f85ef5e5ac9c5c4a8876d6b7870e2fff51d7b1676d: Status 404 returned error can't find the container with id 53c9daf1470dde811a1820f85ef5e5ac9c5c4a8876d6b7870e2fff51d7b1676d
Nov 03 19:46:18.826207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.826159    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.826267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.826242    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.826316 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.826273    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.833907 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.833877    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:18.871063 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:18.871037    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04530f22_b013_40fe_8390_81a0ea009267.slice/crio-c527cfdd4ee8d25d7f92a34640cd730f3e1fa1c18cb4a1bbe2fb99ebfac83cf1 WatchSource:0}: Error finding container c527cfdd4ee8d25d7f92a34640cd730f3e1fa1c18cb4a1bbe2fb99ebfac83cf1: Status 404 returned error can't find the container with id c527cfdd4ee8d25d7f92a34640cd730f3e1fa1c18cb4a1bbe2fb99ebfac83cf1
Nov 03 19:46:18.875022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.874999    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"]
Nov 03 19:46:18.875239 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.875207    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.875314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.875289    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.875383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:18.875333    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:19.052748 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.052689    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:19.133032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.133010    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"]
Nov 03 19:46:19.223807 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.223765    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" event={"ID":"04530f22-b013-40fe-8390-81a0ea009267","Type":"ContainerStarted","Data":"36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a"}
Nov 03 19:46:19.224400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.223839    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" event={"ID":"04530f22-b013-40fe-8390-81a0ea009267","Type":"ContainerStarted","Data":"c527cfdd4ee8d25d7f92a34640cd730f3e1fa1c18cb4a1bbe2fb99ebfac83cf1"}
Nov 03 19:46:19.226388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.226358    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg" event={"ID":"24c79918-75c1-40f5-bbfe-5fdcf99f76c5","Type":"ContainerDied","Data":"f98ae41767f984e896e47713e9fe1e2bd885051b1a638bca8ecf99fec1e1138b"}
Nov 03 19:46:19.226509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.226415    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f98ae41767f984e896e47713e9fe1e2bd885051b1a638bca8ecf99fec1e1138b"
Nov 03 19:46:19.226509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.226493    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"
Nov 03 19:46:19.230771 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.230739    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt" event={"ID":"8bb955c9-1bfe-49de-99b2-2f3b36f2a066","Type":"ContainerStarted","Data":"7078c720389a72437ae67b52608d1def1432700312eda9fab1d7e2d2a6ed0b2b"}
Nov 03 19:46:19.233014 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.232991    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" event={"ID":"7e1a97c8-d12c-46eb-a635-e450998128d5","Type":"ContainerStarted","Data":"4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff"}
Nov 03 19:46:19.233156 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.233142    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" event={"ID":"7e1a97c8-d12c-46eb-a635-e450998128d5","Type":"ContainerStarted","Data":"53c9daf1470dde811a1820f85ef5e5ac9c5c4a8876d6b7870e2fff51d7b1676d"}
Nov 03 19:46:19.249865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.249802    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podStartSLOduration=1.249782046 podStartE2EDuration="1.249782046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:19.244396329 +0000 UTC m=+892.514519626" watchObservedRunningTime="2026-11-03 19:46:19.249782046 +0000 UTC m=+892.519905342"
Nov 03 19:46:19.255116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.254033    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" event={"ID":"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2","Type":"ContainerStarted","Data":"82fefa89998d203254f5768fff02651ed935b615ecb2644d6ef2086f43341258"}
Nov 03 19:46:19.255116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.254106    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" event={"ID":"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2","Type":"ContainerStarted","Data":"53d9a3bc0ed466cc69b481792177381d18a625d3875c93a7de3947dd9faf1d19"}
Nov 03 19:46:19.273686 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.273628    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podStartSLOduration=1.273607865 podStartE2EDuration="1.273607865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:19.266405657 +0000 UTC m=+892.536528954" watchObservedRunningTime="2026-11-03 19:46:19.273607865 +0000 UTC m=+892.543731162"
Nov 03 19:46:19.301191 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.301058    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podStartSLOduration=2.3010356339999998 podStartE2EDuration="2.301035634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:19.298541516 +0000 UTC m=+892.568664844" watchObservedRunningTime="2026-11-03 19:46:19.301035634 +0000 UTC m=+892.571158932"
Nov 03 19:46:19.532710 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.532645    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:19.534403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.534371    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:19.534549 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.534432    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:19.575353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.575264    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:19.768033 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.767775    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:19.973000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.972965    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:19.973379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.973355    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:19.973528 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:19.973506    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:20.323143 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:20.323100    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt" event={"ID":"8bb955c9-1bfe-49de-99b2-2f3b36f2a066","Type":"ContainerStarted","Data":"ee051535539bf1382884b1a5d23e4a576d5d47f08721cb9720874001c0c81287"}
Nov 03 19:46:20.389466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:20.375331    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt" podStartSLOduration=2.375305223 podStartE2EDuration="2.375305223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:20.350485042 +0000 UTC m=+893.620608338" watchObservedRunningTime="2026-11-03 19:46:20.375305223 +0000 UTC m=+893.645428518"
Nov 03 19:46:20.533162 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:20.533130    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:20.533343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:20.533215    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:20.589265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:20.587905    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:20.589265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:20.587977    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:20.615833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:20.615792    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:20.751509 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:46:20.751472    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:46:20.987199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:20.971652    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:20.987199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:20.971723    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:21.334323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:21.334242    4531 generic.go:356] "Generic (PLEG): container finished" podID="8bb955c9-1bfe-49de-99b2-2f3b36f2a066" containerID="ee051535539bf1382884b1a5d23e4a576d5d47f08721cb9720874001c0c81287" exitCode=0
Nov 03 19:46:21.334797 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:21.334325    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt" event={"ID":"8bb955c9-1bfe-49de-99b2-2f3b36f2a066","Type":"ContainerDied","Data":"ee051535539bf1382884b1a5d23e4a576d5d47f08721cb9720874001c0c81287"}
Nov 03 19:46:21.528448 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:21.528402    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:21.582117 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:21.582081    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:21.582480 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:21.582451    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:21.583088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:21.582373    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:21.583264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:21.583246    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:21.979198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:21.974500    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:21.979198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:21.974565    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:22.532788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:22.532756    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:22.533148 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:22.532807    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:22.605740 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:22.586575    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:22.605740 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:22.586635    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:22.717420 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:22.717361    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:23.174208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.161876    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:23.174208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.161935    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:23.385196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.382477    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:23.528206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.508882    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-kube-api-access-jcfvz\" (UniqueName: \"kubernetes.io/projected/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-kube-api-access-jcfvz\") pod \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") "
Nov 03 19:46:23.528206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.509114    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-xks-ca\") pod \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") "
Nov 03 19:46:23.528206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.509205    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-openshift-ca\") pod \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\" (UID: \"8bb955c9-1bfe-49de-99b2-2f3b36f2a066\") "
Nov 03 19:46:23.589078 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.589034    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-openshift-ca" pod "8bb955c9-1bfe-49de-99b2-2f3b36f2a066" (UID: "8bb955c9-1bfe-49de-99b2-2f3b36f2a066"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:23.683267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.663328    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:23.683267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.663393    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:23.683267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.663325    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:23.683267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.663812    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:23.683267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.665698    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:23.739547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.739500    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt" event={"ID":"8bb955c9-1bfe-49de-99b2-2f3b36f2a066","Type":"ContainerDied","Data":"7078c720389a72437ae67b52608d1def1432700312eda9fab1d7e2d2a6ed0b2b"}
Nov 03 19:46:23.763307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.763267    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7078c720389a72437ae67b52608d1def1432700312eda9fab1d7e2d2a6ed0b2b"
Nov 03 19:46:23.763506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.742582    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"
Nov 03 19:46:23.802309 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.802265    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-xks-ca" pod "8bb955c9-1bfe-49de-99b2-2f3b36f2a066" (UID: "8bb955c9-1bfe-49de-99b2-2f3b36f2a066"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:23.874451 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.874404    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-kube-api-access-jcfvz" pod "8bb955c9-1bfe-49de-99b2-2f3b36f2a066" (UID: "8bb955c9-1bfe-49de-99b2-2f3b36f2a066"). InnerVolumeSpecName "kube-api-access-jcfvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:23.915206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:23.893884    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:24.072263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.027258    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcfvz\" (UniqueName: \"kubernetes.io/projected/8bb955c9-1bfe-49de-99b2-2f3b36f2a066-kube-api-access-jcfvz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:24.182122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.182082    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:24.216325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.216293    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:24.217653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.215758    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:46:24.262319 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.262269    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:24.264723 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.262764    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:24.301391 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.301344    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:24.322830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.322716    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:24.338996 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.338945    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:24.395355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.395286    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:24.554703 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.554664    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:24.636262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.617297    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:24.636262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.589348    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Nov 03 19:46:24.636262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.589286    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:24.636262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.589763    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:24.636262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.617439    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:24.636262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.617593    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:46:24.691322 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.691270    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:46:24.692025 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.691987    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:24.699246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.699213    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:24.758206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.755482    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:24.823345 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.823287    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:24.979252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.978388    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:24.979252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:24.978449    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:25.107197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.089263    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.229197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.228274    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.229197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.229102    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:46:25.279709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.279599    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.288938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.288899    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.313199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.305309    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.331219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.322548    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:46:25.331219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.322659    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:46:25.331219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.322705    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:46:25.331219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.323779    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.331219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.325605    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:46:25.363217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.361390    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:46:25.363217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.362199    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:46:25.410888 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.407328    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.410888 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.407966    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:46:25.684378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.684336    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:46:25.729347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.714903    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:46:25.729347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.716259    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:25.729347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.716307    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:25.729347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.716382    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:25.729347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.716408    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:25.740145 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.740037    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.787210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.746392    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.787210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.746547    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.787210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.746810    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:46:25.787210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.746874    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:46:26.074261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.787693    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.074261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.789426    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:46:26.074261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.848851    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:46:26.074261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:25.876613    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:46:26.074261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.037120    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:26.074261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.037194    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:26.307934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.307882    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:46:26.343569 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.343471    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:46:26.352314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.352264    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.380164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.380122    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:46:26.407358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.407315    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:46:26.408809 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.408777    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:46:26.431061 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.431020    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.432510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.432477    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:46:26.709876 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.709188    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.776315 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.749498    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.888308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.750249    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.905193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.814530    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.905193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.888722    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:26.905193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:26.888764    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.044255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.040654    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:27.044255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.040712    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:27.044255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.040790    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:27.044255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.040812    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:27.044255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.041330    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:46:27.068199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.061488    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:27.068199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.061560    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.266198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.263060    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:27.770241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.746486    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.770894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.499286    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.835722 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.835677    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.956277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.514950    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.956660 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.531017    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:46:27.957014 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.578401    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:27.957341 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.677973    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.957617 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.514912    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.957867 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.438287    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.997201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.581629    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:27.997201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.977270    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.997201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.906498    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:46:27.997201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.906319    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.997201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.706793    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:27.997201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.978600    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.997201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.905960    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:27.997201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.978649    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:27.997201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:27.907437    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:46:28.031975 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.031872    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:28.033346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.033291    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:28.037424 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.037397    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:28.037616 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.037591    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:28.096197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.069137    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:28.133651 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.083426    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:28.140204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.133921    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:28.611200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.605282    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:28.802198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.763801    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:28.802198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.779308    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:46:28.841359 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.841304    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:28.909236 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:46:28.908829    4531 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.055s"
Nov 03 19:46:28.931068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.931029    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:28.956334 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:28.956282    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:29.044339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.044288    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Nov 03 19:46:29.063199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.059560    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.067197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.063751    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.084200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.068404    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.085608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.085568    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.088016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.086544    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.088194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.086586    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.088503 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.086602    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.106245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.093528    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.106245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.094733    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.106245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.095065    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.163529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.109823    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:29.197567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.182243    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:29.197567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.164714    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:29.197567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.117221    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.197567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.169262    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.544824 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.544731    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:29.545090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.545052    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:29.592876 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.592835    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:29.593221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.593154    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:29.789642 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.789595    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.987197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.986614    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:29.987197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:29.986674    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:30.054008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.053951    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:30.073225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.070524    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:30.073225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.071366    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:30.085650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.085609    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:30.088293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.088258    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:30.112196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.092559    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:30.112196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.092841    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:30.112196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.101955    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:46:30.112196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.102317    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:46:30.112196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.102550    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:46:30.194951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.194889    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:30.195601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.195564    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:46:30.436104 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.436074    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:46:30.517575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.515447    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:46:30.526735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.526709    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:46:30.533212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.531910    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:46:30.533212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.532853    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:30.533212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.532895    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:30.582194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.577901    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:30.582194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.577956    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:30.590442 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.590416    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:46:30.625219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.622410    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:46:30.726244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.724903    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:46:30.738192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.738150    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:46:30.789223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.780360    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:46:30.790852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.790816    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:30.834074 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.834039    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:46:30.837001 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.836979    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:46:30.959944 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.959737    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:46:30.960234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.960218    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:46:30.992685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.992607    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 19:46:30.993228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:30.993197    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 19:46:31.033234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.033201    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:46:31.532727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.532682    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 19:46:31.532902 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.532752    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 19:46:31.575752 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.575717    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 19:46:31.575935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.575778    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 19:46:31.726128 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.726094    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:31.945913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.945881    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"]
Nov 03 19:46:31.957228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.957200    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8bb955c9-1bfe-49de-99b2-2f3b36f2a066" containerName="revoke-keys"
Nov 03 19:46:31.957412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.957396    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb955c9-1bfe-49de-99b2-2f3b36f2a066" containerName="revoke-keys"
Nov 03 19:46:31.957724 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.957707    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb955c9-1bfe-49de-99b2-2f3b36f2a066" containerName="revoke-keys"
Nov 03 19:46:31.995377 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.995335    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"]
Nov 03 19:46:31.996141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:31.996106    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:32.005357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.005302    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:46:32.005624 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.005440    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:46:32.021203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.011282    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"]
Nov 03 19:46:32.050226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.030902    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"]
Nov 03 19:46:32.050226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.031040    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:32.150960 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.148394    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7530c3a7-qdfzz\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:32.150960 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.148466    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9fjb\" (UniqueName: \"kubernetes.io/projected/4f97d12e-2681-4437-aac9-f3f1dd611bf6-kube-api-access-l9fjb\") pod \"maas-api-revoke-sub-78d7c820-j72q2\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:32.150960 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.148528    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmvqs\" (UniqueName: \"kubernetes.io/projected/79b07052-9a9f-43a5-a99a-47e3a5d13826-kube-api-access-gmvqs\") pod \"maas-api-revoke-sub-7530c3a7-qdfzz\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:32.150960 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.148597    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78d7c820-j72q2\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:32.150960 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.148767    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78d7c820-j72q2\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:32.150960 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.148811    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7530c3a7-qdfzz\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:32.251502 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.251389    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78d7c820-j72q2\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:32.252661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.252539    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7530c3a7-qdfzz\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:32.253219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.253199    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78d7c820-j72q2\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:32.254397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.253165    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7530c3a7-qdfzz\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:32.254712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.254693    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9fjb\" (UniqueName: \"kubernetes.io/projected/4f97d12e-2681-4437-aac9-f3f1dd611bf6-kube-api-access-l9fjb\") pod \"maas-api-revoke-sub-78d7c820-j72q2\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:32.254988 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.254970    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmvqs\" (UniqueName: \"kubernetes.io/projected/79b07052-9a9f-43a5-a99a-47e3a5d13826-kube-api-access-gmvqs\") pod \"maas-api-revoke-sub-7530c3a7-qdfzz\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:32.255412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.255392    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78d7c820-j72q2\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:32.256933 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.256912    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7530c3a7-qdfzz\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:32.262595 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.258597    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7530c3a7-qdfzz\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:32.263539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.263508    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78d7c820-j72q2\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:32.273284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.273248    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmvqs\" (UniqueName: \"kubernetes.io/projected/79b07052-9a9f-43a5-a99a-47e3a5d13826-kube-api-access-gmvqs\") pod \"maas-api-revoke-sub-7530c3a7-qdfzz\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:32.278957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.278932    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9fjb\" (UniqueName: \"kubernetes.io/projected/4f97d12e-2681-4437-aac9-f3f1dd611bf6-kube-api-access-l9fjb\") pod \"maas-api-revoke-sub-78d7c820-j72q2\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:32.448814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.448775    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:32.456422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.456392    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:32.540247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.538501    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:32.540247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.539953    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:32.540752 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.540703    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:46:32.658803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.658750    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:32.921107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.921072    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"]
Nov 03 19:46:32.976909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.974242    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:32.976909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.975703    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:32.980919 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.980884    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:32.985002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.984968    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:32.985151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.985024    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:32.985776 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:32.985717    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:46:33.005514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:33.005477    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"]
Nov 03 19:46:33.177836 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:33.177798    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2" event={"ID":"4f97d12e-2681-4437-aac9-f3f1dd611bf6","Type":"ContainerStarted","Data":"8ec44e61fa6bca7fbbb35111b393457dee1d91e4dc6f6b7632d50f14680de063"}
Nov 03 19:46:33.181808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:33.181773    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz" event={"ID":"79b07052-9a9f-43a5-a99a-47e3a5d13826","Type":"ContainerStarted","Data":"5578c8be573683038c135e93643baf50820d5af68a12a76e3eda777fac1ca55b"}
Nov 03 19:46:33.529711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:33.529675    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:33.868044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:33.868012    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:33.869505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:33.869480    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 19:46:34.191473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.191434    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2" event={"ID":"4f97d12e-2681-4437-aac9-f3f1dd611bf6","Type":"ContainerStarted","Data":"2006cc46390ab1b2e044c64e36cfd398534a4688495cd14f55e3181c7aec2177"}
Nov 03 19:46:34.193166 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.193136    4531 generic.go:356] "Generic (PLEG): container finished" podID="79b07052-9a9f-43a5-a99a-47e3a5d13826" containerID="1cf700c12ef41f0b5d53b293b021cadf954c45e735081b8f3c81b7533cac6a6d" exitCode=0
Nov 03 19:46:34.193339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.193199    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz" event={"ID":"79b07052-9a9f-43a5-a99a-47e3a5d13826","Type":"ContainerDied","Data":"1cf700c12ef41f0b5d53b293b021cadf954c45e735081b8f3c81b7533cac6a6d"}
Nov 03 19:46:34.211120 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.211073    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2" podStartSLOduration=3.211059 podStartE2EDuration="3.211059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:34.205220132 +0000 UTC m=+907.475343442" watchObservedRunningTime="2026-11-03 19:46:34.211059 +0000 UTC m=+907.481182295"
Nov 03 19:46:34.246885 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.246800    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:34.394402 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.390337    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:34.428195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.423443    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"]
Nov 03 19:46:34.473644 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.466295    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"]
Nov 03 19:46:34.473644 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.466435    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:34.523914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.523881    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfjgb\" (UniqueName: \"kubernetes.io/projected/1ba659e9-0e04-4cd1-87dd-45610cfcafed-kube-api-access-zfjgb\") pod \"maas-api-revoke-sub-04da2a04-z7vpq\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:34.524085 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.524018    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04da2a04-z7vpq\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:34.524162 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.524146    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04da2a04-z7vpq\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:34.625685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.625599    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfjgb\" (UniqueName: \"kubernetes.io/projected/1ba659e9-0e04-4cd1-87dd-45610cfcafed-kube-api-access-zfjgb\") pod \"maas-api-revoke-sub-04da2a04-z7vpq\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:34.625836 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.625753    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04da2a04-z7vpq\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:34.625892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.625860    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04da2a04-z7vpq\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:34.626558 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.626528    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04da2a04-z7vpq\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:34.628207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.628165    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04da2a04-z7vpq\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:34.632788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.632765    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfjgb\" (UniqueName: \"kubernetes.io/projected/1ba659e9-0e04-4cd1-87dd-45610cfcafed-kube-api-access-zfjgb\") pod \"maas-api-revoke-sub-04da2a04-z7vpq\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:34.798367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:34.798334    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:35.096474 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:35.096438    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba659e9_0e04_4cd1_87dd_45610cfcafed.slice/crio-b9cf24372a4e573afa641137826100dba030e82a27bccd33db33dada3f436b05 WatchSource:0}: Error finding container b9cf24372a4e573afa641137826100dba030e82a27bccd33db33dada3f436b05: Status 404 returned error can't find the container with id b9cf24372a4e573afa641137826100dba030e82a27bccd33db33dada3f436b05
Nov 03 19:46:35.098332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.098302    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"]
Nov 03 19:46:35.202873 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.202832    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq" event={"ID":"1ba659e9-0e04-4cd1-87dd-45610cfcafed","Type":"ContainerStarted","Data":"b9cf24372a4e573afa641137826100dba030e82a27bccd33db33dada3f436b05"}
Nov 03 19:46:35.204594 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.204566    4531 generic.go:356] "Generic (PLEG): container finished" podID="4f97d12e-2681-4437-aac9-f3f1dd611bf6" containerID="2006cc46390ab1b2e044c64e36cfd398534a4688495cd14f55e3181c7aec2177" exitCode=0
Nov 03 19:46:35.204721 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.204647    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2" event={"ID":"4f97d12e-2681-4437-aac9-f3f1dd611bf6","Type":"ContainerDied","Data":"2006cc46390ab1b2e044c64e36cfd398534a4688495cd14f55e3181c7aec2177"}
Nov 03 19:46:35.279197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.275774    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:35.560564 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.560531    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:35.671440 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.671405    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-openshift-ca\") pod \"79b07052-9a9f-43a5-a99a-47e3a5d13826\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") "
Nov 03 19:46:35.671684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.671670    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-xks-ca\") pod \"79b07052-9a9f-43a5-a99a-47e3a5d13826\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") "
Nov 03 19:46:35.671773 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.671705    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b07052-9a9f-43a5-a99a-47e3a5d13826-kube-api-access-gmvqs\" (UniqueName: \"kubernetes.io/projected/79b07052-9a9f-43a5-a99a-47e3a5d13826-kube-api-access-gmvqs\") pod \"79b07052-9a9f-43a5-a99a-47e3a5d13826\" (UID: \"79b07052-9a9f-43a5-a99a-47e3a5d13826\") "
Nov 03 19:46:35.671839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.671813    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-openshift-ca" pod "79b07052-9a9f-43a5-a99a-47e3a5d13826" (UID: "79b07052-9a9f-43a5-a99a-47e3a5d13826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:35.672337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.672315    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:35.673910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.673887    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-xks-ca" pod "79b07052-9a9f-43a5-a99a-47e3a5d13826" (UID: "79b07052-9a9f-43a5-a99a-47e3a5d13826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:35.673910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.673896    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b07052-9a9f-43a5-a99a-47e3a5d13826-kube-api-access-gmvqs" pod "79b07052-9a9f-43a5-a99a-47e3a5d13826" (UID: "79b07052-9a9f-43a5-a99a-47e3a5d13826"). InnerVolumeSpecName "kube-api-access-gmvqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:35.773936 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.773856    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b07052-9a9f-43a5-a99a-47e3a5d13826-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:35.773936 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:35.773887    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmvqs\" (UniqueName: \"kubernetes.io/projected/79b07052-9a9f-43a5-a99a-47e3a5d13826-kube-api-access-gmvqs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:36.076254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.074074    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:36.215606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.215573    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"
Nov 03 19:46:36.216051 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.215604    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz" event={"ID":"79b07052-9a9f-43a5-a99a-47e3a5d13826","Type":"ContainerDied","Data":"5578c8be573683038c135e93643baf50820d5af68a12a76e3eda777fac1ca55b"}
Nov 03 19:46:36.216051 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.215654    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5578c8be573683038c135e93643baf50820d5af68a12a76e3eda777fac1ca55b"
Nov 03 19:46:36.217421 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.217397    4531 generic.go:356] "Generic (PLEG): container finished" podID="1ba659e9-0e04-4cd1-87dd-45610cfcafed" containerID="e76efef59cfc1f5c9efaf53b67992e5c96111de47729d4053c536ac5bd7049c2" exitCode=0
Nov 03 19:46:36.217570 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.217461    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq" event={"ID":"1ba659e9-0e04-4cd1-87dd-45610cfcafed","Type":"ContainerDied","Data":"e76efef59cfc1f5c9efaf53b67992e5c96111de47729d4053c536ac5bd7049c2"}
Nov 03 19:46:36.352864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.352835    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:36.490255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.490218    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-xks-ca\") pod \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") "
Nov 03 19:46:36.490492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.490476    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-openshift-ca\") pod \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") "
Nov 03 19:46:36.490592 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.490513    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f97d12e-2681-4437-aac9-f3f1dd611bf6-kube-api-access-l9fjb\" (UniqueName: \"kubernetes.io/projected/4f97d12e-2681-4437-aac9-f3f1dd611bf6-kube-api-access-l9fjb\") pod \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\" (UID: \"4f97d12e-2681-4437-aac9-f3f1dd611bf6\") "
Nov 03 19:46:36.490924 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.490896    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-openshift-ca" pod "4f97d12e-2681-4437-aac9-f3f1dd611bf6" (UID: "4f97d12e-2681-4437-aac9-f3f1dd611bf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:36.491544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.491518    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:36.492647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.492622    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-xks-ca" pod "4f97d12e-2681-4437-aac9-f3f1dd611bf6" (UID: "4f97d12e-2681-4437-aac9-f3f1dd611bf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:36.492771 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.492754    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f97d12e-2681-4437-aac9-f3f1dd611bf6-kube-api-access-l9fjb" pod "4f97d12e-2681-4437-aac9-f3f1dd611bf6" (UID: "4f97d12e-2681-4437-aac9-f3f1dd611bf6"). InnerVolumeSpecName "kube-api-access-l9fjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:36.592852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.592747    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9fjb\" (UniqueName: \"kubernetes.io/projected/4f97d12e-2681-4437-aac9-f3f1dd611bf6-kube-api-access-l9fjb\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:36.592852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.592783    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f97d12e-2681-4437-aac9-f3f1dd611bf6-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:36.975196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:36.966003    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:37.074488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.074455    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"]
Nov 03 19:46:37.075159 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.075145    4531 cpu_manager.go:414] "containerMap: removing container" podUID="79b07052-9a9f-43a5-a99a-47e3a5d13826" containerName="revoke-keys"
Nov 03 19:46:37.075212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.075160    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b07052-9a9f-43a5-a99a-47e3a5d13826" containerName="revoke-keys"
Nov 03 19:46:37.075265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.075235    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4f97d12e-2681-4437-aac9-f3f1dd611bf6" containerName="revoke-keys"
Nov 03 19:46:37.075265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.075243    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f97d12e-2681-4437-aac9-f3f1dd611bf6" containerName="revoke-keys"
Nov 03 19:46:37.075385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.075375    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f97d12e-2681-4437-aac9-f3f1dd611bf6" containerName="revoke-keys"
Nov 03 19:46:37.075417 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.075391    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b07052-9a9f-43a5-a99a-47e3a5d13826" containerName="revoke-keys"
Nov 03 19:46:37.079968 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.079940    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:37.086787 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.086041    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"]
Nov 03 19:46:37.216770 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.216678    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11258e8f-c28tl\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:37.217159 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.216775    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11258e8f-c28tl\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:37.217159 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.216967    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2sc\" (UniqueName: \"kubernetes.io/projected/aa4cacd3-f23c-48bb-b303-1d0773d26915-kube-api-access-6m2sc\") pod \"maas-api-revoke-sub-11258e8f-c28tl\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:37.226842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.226816    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"
Nov 03 19:46:37.227021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.226814    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2" event={"ID":"4f97d12e-2681-4437-aac9-f3f1dd611bf6","Type":"ContainerDied","Data":"8ec44e61fa6bca7fbbb35111b393457dee1d91e4dc6f6b7632d50f14680de063"}
Nov 03 19:46:37.227021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.226977    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ec44e61fa6bca7fbbb35111b393457dee1d91e4dc6f6b7632d50f14680de063"
Nov 03 19:46:37.319383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.319344    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2sc\" (UniqueName: \"kubernetes.io/projected/aa4cacd3-f23c-48bb-b303-1d0773d26915-kube-api-access-6m2sc\") pod \"maas-api-revoke-sub-11258e8f-c28tl\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:37.319650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.319627    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11258e8f-c28tl\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:37.319782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.319753    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11258e8f-c28tl\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:37.320739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.320712    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11258e8f-c28tl\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:37.323000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.322975    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11258e8f-c28tl\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:37.327491 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.327464    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2sc\" (UniqueName: \"kubernetes.io/projected/aa4cacd3-f23c-48bb-b303-1d0773d26915-kube-api-access-6m2sc\") pod \"maas-api-revoke-sub-11258e8f-c28tl\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:37.401163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.401132    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:37.403769 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.403743    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:37.523721 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.523679    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-openshift-ca\") pod \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") "
Nov 03 19:46:37.524044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.524018    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba659e9-0e04-4cd1-87dd-45610cfcafed-kube-api-access-zfjgb\" (UniqueName: \"kubernetes.io/projected/1ba659e9-0e04-4cd1-87dd-45610cfcafed-kube-api-access-zfjgb\") pod \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") "
Nov 03 19:46:37.524161 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.524146    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-xks-ca\") pod \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\" (UID: \"1ba659e9-0e04-4cd1-87dd-45610cfcafed\") "
Nov 03 19:46:37.526425 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.526384    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-openshift-ca" pod "1ba659e9-0e04-4cd1-87dd-45610cfcafed" (UID: "1ba659e9-0e04-4cd1-87dd-45610cfcafed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:37.528045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.528016    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba659e9-0e04-4cd1-87dd-45610cfcafed-kube-api-access-zfjgb" pod "1ba659e9-0e04-4cd1-87dd-45610cfcafed" (UID: "1ba659e9-0e04-4cd1-87dd-45610cfcafed"). InnerVolumeSpecName "kube-api-access-zfjgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:37.528314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.528287    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-xks-ca" pod "1ba659e9-0e04-4cd1-87dd-45610cfcafed" (UID: "1ba659e9-0e04-4cd1-87dd-45610cfcafed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:37.626735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.626696    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfjgb\" (UniqueName: \"kubernetes.io/projected/1ba659e9-0e04-4cd1-87dd-45610cfcafed-kube-api-access-zfjgb\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:37.626735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.626733    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:37.626735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.626743    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba659e9-0e04-4cd1-87dd-45610cfcafed-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:37.702828 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:37.702788    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4cacd3_f23c_48bb_b303_1d0773d26915.slice/crio-ce8623512535f082c6de0d639813e9e1fc8348ef16cc40a859d54b65e8bb7840 WatchSource:0}: Error finding container ce8623512535f082c6de0d639813e9e1fc8348ef16cc40a859d54b65e8bb7840: Status 404 returned error can't find the container with id ce8623512535f082c6de0d639813e9e1fc8348ef16cc40a859d54b65e8bb7840
Nov 03 19:46:37.703892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.703867    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"]
Nov 03 19:46:37.842196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:37.840771    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:38.236045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:38.236014    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"
Nov 03 19:46:38.236534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:38.236051    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq" event={"ID":"1ba659e9-0e04-4cd1-87dd-45610cfcafed","Type":"ContainerDied","Data":"b9cf24372a4e573afa641137826100dba030e82a27bccd33db33dada3f436b05"}
Nov 03 19:46:38.236534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:38.236101    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9cf24372a4e573afa641137826100dba030e82a27bccd33db33dada3f436b05"
Nov 03 19:46:38.237953 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:38.237929    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl" event={"ID":"aa4cacd3-f23c-48bb-b303-1d0773d26915","Type":"ContainerStarted","Data":"aa903fa27798fb32d84009cb02b3956f87643123c504f48e019272d32d02921f"}
Nov 03 19:46:38.238095 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:38.237979    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl" event={"ID":"aa4cacd3-f23c-48bb-b303-1d0773d26915","Type":"ContainerStarted","Data":"ce8623512535f082c6de0d639813e9e1fc8348ef16cc40a859d54b65e8bb7840"}
Nov 03 19:46:38.251978 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:38.251928    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl" podStartSLOduration=1.251913257 podStartE2EDuration="1.251913257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:38.250760756 +0000 UTC m=+911.520884053" watchObservedRunningTime="2026-11-03 19:46:38.251913257 +0000 UTC m=+911.522036554"
Nov 03 19:46:38.577266 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:38.576957    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:39.248858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:39.248816    4531 generic.go:356] "Generic (PLEG): container finished" podID="aa4cacd3-f23c-48bb-b303-1d0773d26915" containerID="aa903fa27798fb32d84009cb02b3956f87643123c504f48e019272d32d02921f" exitCode=0
Nov 03 19:46:39.249293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:39.248904    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl" event={"ID":"aa4cacd3-f23c-48bb-b303-1d0773d26915","Type":"ContainerDied","Data":"aa903fa27798fb32d84009cb02b3956f87643123c504f48e019272d32d02921f"}
Nov 03 19:46:39.398197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:39.398024    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:40.125204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.121711    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:40.425629 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.425603    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:40.511448 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.511409    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4cacd3-f23c-48bb-b303-1d0773d26915-kube-api-access-6m2sc\" (UniqueName: \"kubernetes.io/projected/aa4cacd3-f23c-48bb-b303-1d0773d26915-kube-api-access-6m2sc\") pod \"aa4cacd3-f23c-48bb-b303-1d0773d26915\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") "
Nov 03 19:46:40.511650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.511607    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-xks-ca\") pod \"aa4cacd3-f23c-48bb-b303-1d0773d26915\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") "
Nov 03 19:46:40.511726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.511714    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-openshift-ca\") pod \"aa4cacd3-f23c-48bb-b303-1d0773d26915\" (UID: \"aa4cacd3-f23c-48bb-b303-1d0773d26915\") "
Nov 03 19:46:40.512259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.512226    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-openshift-ca" pod "aa4cacd3-f23c-48bb-b303-1d0773d26915" (UID: "aa4cacd3-f23c-48bb-b303-1d0773d26915"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:40.512653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.512629    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:40.513868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.513841    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-xks-ca" pod "aa4cacd3-f23c-48bb-b303-1d0773d26915" (UID: "aa4cacd3-f23c-48bb-b303-1d0773d26915"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:40.513868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.513849    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4cacd3-f23c-48bb-b303-1d0773d26915-kube-api-access-6m2sc" pod "aa4cacd3-f23c-48bb-b303-1d0773d26915" (UID: "aa4cacd3-f23c-48bb-b303-1d0773d26915"). InnerVolumeSpecName "kube-api-access-6m2sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:40.614736 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.614699    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4cacd3-f23c-48bb-b303-1d0773d26915-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:40.614736 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.614733    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m2sc\" (UniqueName: \"kubernetes.io/projected/aa4cacd3-f23c-48bb-b303-1d0773d26915-kube-api-access-6m2sc\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:40.852248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:40.846409    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:41.255248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.255210    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"]
Nov 03 19:46:41.255965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.255943    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1ba659e9-0e04-4cd1-87dd-45610cfcafed" containerName="revoke-keys"
Nov 03 19:46:41.255965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.255962    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba659e9-0e04-4cd1-87dd-45610cfcafed" containerName="revoke-keys"
Nov 03 19:46:41.256135 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.255984    4531 cpu_manager.go:414] "containerMap: removing container" podUID="aa4cacd3-f23c-48bb-b303-1d0773d26915" containerName="revoke-keys"
Nov 03 19:46:41.256135 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.255990    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4cacd3-f23c-48bb-b303-1d0773d26915" containerName="revoke-keys"
Nov 03 19:46:41.256256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.256157    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4cacd3-f23c-48bb-b303-1d0773d26915" containerName="revoke-keys"
Nov 03 19:46:41.256256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.256199    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba659e9-0e04-4cd1-87dd-45610cfcafed" containerName="revoke-keys"
Nov 03 19:46:41.260686 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.260661    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:41.266482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.266453    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"]
Nov 03 19:46:41.271450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.270393    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"
Nov 03 19:46:41.271450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.270391    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl" event={"ID":"aa4cacd3-f23c-48bb-b303-1d0773d26915","Type":"ContainerDied","Data":"ce8623512535f082c6de0d639813e9e1fc8348ef16cc40a859d54b65e8bb7840"}
Nov 03 19:46:41.271450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.270602    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce8623512535f082c6de0d639813e9e1fc8348ef16cc40a859d54b65e8bb7840"
Nov 03 19:46:41.344704 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.344668    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b263cacf-zr2x7\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:41.344926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.344902    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zrhh\" (UniqueName: \"kubernetes.io/projected/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-kube-api-access-4zrhh\") pod \"maas-api-revoke-sub-b263cacf-zr2x7\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:41.345075 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.345058    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b263cacf-zr2x7\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:41.446735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.446698    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b263cacf-zr2x7\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:41.447291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.446850    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b263cacf-zr2x7\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:41.447291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.446974    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrhh\" (UniqueName: \"kubernetes.io/projected/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-kube-api-access-4zrhh\") pod \"maas-api-revoke-sub-b263cacf-zr2x7\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:41.447764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.447739    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b263cacf-zr2x7\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:41.449468 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.449445    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b263cacf-zr2x7\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:41.456496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.456476    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zrhh\" (UniqueName: \"kubernetes.io/projected/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-kube-api-access-4zrhh\") pod \"maas-api-revoke-sub-b263cacf-zr2x7\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:41.513007 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.512920    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:41.599397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.584450    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:41.905809 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:41.905774    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9329d4f_1cfe_4868_a5e6_06a841d7dfc2.slice/crio-9e4a2a421672378dfcf42486047718256e6511cf5303f13cb024177f8a6f737d WatchSource:0}: Error finding container 9e4a2a421672378dfcf42486047718256e6511cf5303f13cb024177f8a6f737d: Status 404 returned error can't find the container with id 9e4a2a421672378dfcf42486047718256e6511cf5303f13cb024177f8a6f737d
Nov 03 19:46:41.907822 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:41.907800    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"]
Nov 03 19:46:42.280290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:42.280256    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7" event={"ID":"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2","Type":"ContainerStarted","Data":"b6c5759d4f1e36d4e356d0cb32a88bac712df7f4dc6d78a04078896b8cece468"}
Nov 03 19:46:42.280450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:42.280333    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7" event={"ID":"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2","Type":"ContainerStarted","Data":"9e4a2a421672378dfcf42486047718256e6511cf5303f13cb024177f8a6f737d"}
Nov 03 19:46:42.295630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:42.295575    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7" podStartSLOduration=1.295560446 podStartE2EDuration="1.295560446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:42.292751786 +0000 UTC m=+915.562875084" watchObservedRunningTime="2026-11-03 19:46:42.295560446 +0000 UTC m=+915.565683743"
Nov 03 19:46:42.408207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:42.408075    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:43.244758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.244722    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:43.376252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.367910    4531 generic.go:356] "Generic (PLEG): container finished" podID="d9329d4f-1cfe-4868-a5e6-06a841d7dfc2" containerID="b6c5759d4f1e36d4e356d0cb32a88bac712df7f4dc6d78a04078896b8cece468" exitCode=0
Nov 03 19:46:43.376252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.367991    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7" event={"ID":"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2","Type":"ContainerDied","Data":"b6c5759d4f1e36d4e356d0cb32a88bac712df7f4dc6d78a04078896b8cece468"}
Nov 03 19:46:43.397889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.395115    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"]
Nov 03 19:46:43.400241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.400168    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:43.417956 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.417926    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"]
Nov 03 19:46:43.520996 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.520905    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-871c7fc5-6z8ks\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:43.521151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.521088    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b78rp\" (UniqueName: \"kubernetes.io/projected/33a8b817-0777-4330-b177-9c5bc9f3cc64-kube-api-access-b78rp\") pod \"maas-api-revoke-sub-871c7fc5-6z8ks\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:43.521151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.521135    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-871c7fc5-6z8ks\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:43.622738 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.622706    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-871c7fc5-6z8ks\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:43.622918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.622829    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b78rp\" (UniqueName: \"kubernetes.io/projected/33a8b817-0777-4330-b177-9c5bc9f3cc64-kube-api-access-b78rp\") pod \"maas-api-revoke-sub-871c7fc5-6z8ks\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:43.622918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.622876    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-871c7fc5-6z8ks\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:43.623477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.623451    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-871c7fc5-6z8ks\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:43.625295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.625268    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-871c7fc5-6z8ks\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:43.629770 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.629748    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b78rp\" (UniqueName: \"kubernetes.io/projected/33a8b817-0777-4330-b177-9c5bc9f3cc64-kube-api-access-b78rp\") pod \"maas-api-revoke-sub-871c7fc5-6z8ks\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:43.735301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:43.735266    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:44.020136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.019290    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"]
Nov 03 19:46:44.021586 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:44.021558    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a8b817_0777_4330_b177_9c5bc9f3cc64.slice/crio-debedf10686728bc293f08e3f3a1954f8c624e213da1e3c5b61a6972b9373c1d WatchSource:0}: Error finding container debedf10686728bc293f08e3f3a1954f8c624e213da1e3c5b61a6972b9373c1d: Status 404 returned error can't find the container with id debedf10686728bc293f08e3f3a1954f8c624e213da1e3c5b61a6972b9373c1d
Nov 03 19:46:44.308251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.304082    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:44.398488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.398444    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks" event={"ID":"33a8b817-0777-4330-b177-9c5bc9f3cc64","Type":"ContainerStarted","Data":"debedf10686728bc293f08e3f3a1954f8c624e213da1e3c5b61a6972b9373c1d"}
Nov 03 19:46:44.566023 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.565965    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:44.635365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.635332    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"]
Nov 03 19:46:44.636111 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.636093    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d9329d4f-1cfe-4868-a5e6-06a841d7dfc2" containerName="revoke-keys"
Nov 03 19:46:44.636241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.636112    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9329d4f-1cfe-4868-a5e6-06a841d7dfc2" containerName="revoke-keys"
Nov 03 19:46:44.636307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.636266    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9329d4f-1cfe-4868-a5e6-06a841d7dfc2" containerName="revoke-keys"
Nov 03 19:46:44.649054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.643452    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:44.662192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.659290    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"]
Nov 03 19:46:44.747989 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.747955    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-kube-api-access-4zrhh\" (UniqueName: \"kubernetes.io/projected/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-kube-api-access-4zrhh\") pod \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") "
Nov 03 19:46:44.748158 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.748148    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-xks-ca\") pod \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") "
Nov 03 19:46:44.748257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.748244    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-openshift-ca\") pod \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\" (UID: \"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2\") "
Nov 03 19:46:44.748601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.748574    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-openshift-ca" pod "d9329d4f-1cfe-4868-a5e6-06a841d7dfc2" (UID: "d9329d4f-1cfe-4868-a5e6-06a841d7dfc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:44.748860 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.748835    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40185ba6-kwhtp\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:44.749024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.748996    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8x5\" (UniqueName: \"kubernetes.io/projected/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-kube-api-access-9c8x5\") pod \"maas-api-revoke-sub-40185ba6-kwhtp\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:44.749272 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.749252    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40185ba6-kwhtp\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:44.749428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.749414    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:44.750345 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.750322    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-kube-api-access-4zrhh" pod "d9329d4f-1cfe-4868-a5e6-06a841d7dfc2" (UID: "d9329d4f-1cfe-4868-a5e6-06a841d7dfc2"). InnerVolumeSpecName "kube-api-access-4zrhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:44.750436 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.750402    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-xks-ca" pod "d9329d4f-1cfe-4868-a5e6-06a841d7dfc2" (UID: "d9329d4f-1cfe-4868-a5e6-06a841d7dfc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:44.850548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.850465    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40185ba6-kwhtp\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:44.850717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.850631    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40185ba6-kwhtp\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:44.850717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.850691    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8x5\" (UniqueName: \"kubernetes.io/projected/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-kube-api-access-9c8x5\") pod \"maas-api-revoke-sub-40185ba6-kwhtp\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:44.850891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.850764    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:44.850891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.850874    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zrhh\" (UniqueName: \"kubernetes.io/projected/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2-kube-api-access-4zrhh\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:44.851403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.851376    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40185ba6-kwhtp\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:44.852935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.852914    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40185ba6-kwhtp\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:44.857586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.857561    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8x5\" (UniqueName: \"kubernetes.io/projected/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-kube-api-access-9c8x5\") pod \"maas-api-revoke-sub-40185ba6-kwhtp\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:44.983095 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:44.983065    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:45.170365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.168162    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:45.282489 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.282458    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"]
Nov 03 19:46:45.287473 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:45.287444    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2a5f8f_d8b8_4c8f_91fb_de16cea6c964.slice/crio-77f2660f95a8a3a570621ce648d1efd2a58d978441a5cc69e601f3451abb6011 WatchSource:0}: Error finding container 77f2660f95a8a3a570621ce648d1efd2a58d978441a5cc69e601f3451abb6011: Status 404 returned error can't find the container with id 77f2660f95a8a3a570621ce648d1efd2a58d978441a5cc69e601f3451abb6011
Nov 03 19:46:45.408698 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.408662    4531 generic.go:356] "Generic (PLEG): container finished" podID="33a8b817-0777-4330-b177-9c5bc9f3cc64" containerID="733f275c337c04f85fe43381d59bd49d976e1815f6856bdc41da9333acbfd9d4" exitCode=0
Nov 03 19:46:45.409116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.408745    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks" event={"ID":"33a8b817-0777-4330-b177-9c5bc9f3cc64","Type":"ContainerDied","Data":"733f275c337c04f85fe43381d59bd49d976e1815f6856bdc41da9333acbfd9d4"}
Nov 03 19:46:45.410563 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.410544    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"
Nov 03 19:46:45.410709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.410580    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7" event={"ID":"d9329d4f-1cfe-4868-a5e6-06a841d7dfc2","Type":"ContainerDied","Data":"9e4a2a421672378dfcf42486047718256e6511cf5303f13cb024177f8a6f737d"}
Nov 03 19:46:45.410709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.410629    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e4a2a421672378dfcf42486047718256e6511cf5303f13cb024177f8a6f737d"
Nov 03 19:46:45.412706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.412681    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp" event={"ID":"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964","Type":"ContainerStarted","Data":"77f2660f95a8a3a570621ce648d1efd2a58d978441a5cc69e601f3451abb6011"}
Nov 03 19:46:45.669204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.669123    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:45.710866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.710790    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"]
Nov 03 19:46:45.715854 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.715825    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:45.722621 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.722594    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"]
Nov 03 19:46:45.771455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.771418    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09145056-9jsxp\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:45.771659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.771529    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69tsp\" (UniqueName: \"kubernetes.io/projected/5861fe0f-fe8a-4704-8a51-689479fac3c9-kube-api-access-69tsp\") pod \"maas-api-revoke-sub-09145056-9jsxp\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:45.771742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.771703    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09145056-9jsxp\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:45.827411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.827374    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:45.873382 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.873345    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09145056-9jsxp\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:45.873542 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.873530    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09145056-9jsxp\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:45.873601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.873589    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69tsp\" (UniqueName: \"kubernetes.io/projected/5861fe0f-fe8a-4704-8a51-689479fac3c9-kube-api-access-69tsp\") pod \"maas-api-revoke-sub-09145056-9jsxp\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:45.874220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.874144    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09145056-9jsxp\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:45.875951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.875929    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09145056-9jsxp\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:45.880567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.880546    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69tsp\" (UniqueName: \"kubernetes.io/projected/5861fe0f-fe8a-4704-8a51-689479fac3c9-kube-api-access-69tsp\") pod \"maas-api-revoke-sub-09145056-9jsxp\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:45.982248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:45.981424    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:46.041195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.040893    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:46.324626 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.324577    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:46.325540 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:46.325515    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5861fe0f_fe8a_4704_8a51_689479fac3c9.slice/crio-ebb1fa3a8ddbd506b121482fd03899ae2857c543013dd2278a4acfc7ee4fd5e0 WatchSource:0}: Error finding container ebb1fa3a8ddbd506b121482fd03899ae2857c543013dd2278a4acfc7ee4fd5e0: Status 404 returned error can't find the container with id ebb1fa3a8ddbd506b121482fd03899ae2857c543013dd2278a4acfc7ee4fd5e0
Nov 03 19:46:46.325636 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.325594    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"]
Nov 03 19:46:46.424708 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.424672    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp" event={"ID":"5861fe0f-fe8a-4704-8a51-689479fac3c9","Type":"ContainerStarted","Data":"ebb1fa3a8ddbd506b121482fd03899ae2857c543013dd2278a4acfc7ee4fd5e0"}
Nov 03 19:46:46.426303 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.426270    4531 generic.go:356] "Generic (PLEG): container finished" podID="7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964" containerID="dd025fbe7dd72be54b2e26c747cb23b7f40ad07a3f36594fec879ed5ae878978" exitCode=0
Nov 03 19:46:46.426422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.426360    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp" event={"ID":"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964","Type":"ContainerDied","Data":"dd025fbe7dd72be54b2e26c747cb23b7f40ad07a3f36594fec879ed5ae878978"}
Nov 03 19:46:46.569892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.569868    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:46.592150 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.592123    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a8b817-0777-4330-b177-9c5bc9f3cc64-kube-api-access-b78rp\" (UniqueName: \"kubernetes.io/projected/33a8b817-0777-4330-b177-9c5bc9f3cc64-kube-api-access-b78rp\") pod \"33a8b817-0777-4330-b177-9c5bc9f3cc64\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") "
Nov 03 19:46:46.593113 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.592510    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-xks-ca\") pod \"33a8b817-0777-4330-b177-9c5bc9f3cc64\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") "
Nov 03 19:46:46.593113 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.592615    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-openshift-ca\") pod \"33a8b817-0777-4330-b177-9c5bc9f3cc64\" (UID: \"33a8b817-0777-4330-b177-9c5bc9f3cc64\") "
Nov 03 19:46:46.593113 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.593069    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-openshift-ca" pod "33a8b817-0777-4330-b177-9c5bc9f3cc64" (UID: "33a8b817-0777-4330-b177-9c5bc9f3cc64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:46.593554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.593533    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:46.595250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.595223    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-xks-ca" pod "33a8b817-0777-4330-b177-9c5bc9f3cc64" (UID: "33a8b817-0777-4330-b177-9c5bc9f3cc64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:46.595363 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.595347    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a8b817-0777-4330-b177-9c5bc9f3cc64-kube-api-access-b78rp" pod "33a8b817-0777-4330-b177-9c5bc9f3cc64" (UID: "33a8b817-0777-4330-b177-9c5bc9f3cc64"). InnerVolumeSpecName "kube-api-access-b78rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:46.694832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.694799    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b78rp\" (UniqueName: \"kubernetes.io/projected/33a8b817-0777-4330-b177-9c5bc9f3cc64-kube-api-access-b78rp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:46.694832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.694828    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a8b817-0777-4330-b177-9c5bc9f3cc64-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:46.796619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:46.796261    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:47.313852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.313815    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"]
Nov 03 19:46:47.314604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.314586    4531 cpu_manager.go:414] "containerMap: removing container" podUID="33a8b817-0777-4330-b177-9c5bc9f3cc64" containerName="revoke-keys"
Nov 03 19:46:47.314604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.314603    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a8b817-0777-4330-b177-9c5bc9f3cc64" containerName="revoke-keys"
Nov 03 19:46:47.314712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.314704    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a8b817-0777-4330-b177-9c5bc9f3cc64" containerName="revoke-keys"
Nov 03 19:46:47.318955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.318744    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:47.328025 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.327658    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"]
Nov 03 19:46:47.407326 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.407294    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2ea3ef5-cc5cc\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:47.407510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.407354    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2ea3ef5-cc5cc\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:47.408016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.407982    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgh7n\" (UniqueName: \"kubernetes.io/projected/bd166e04-e728-426b-8222-f251ea015d6f-kube-api-access-lgh7n\") pod \"maas-api-revoke-sub-c2ea3ef5-cc5cc\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:47.437007 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.436972    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks" event={"ID":"33a8b817-0777-4330-b177-9c5bc9f3cc64","Type":"ContainerDied","Data":"debedf10686728bc293f08e3f3a1954f8c624e213da1e3c5b61a6972b9373c1d"}
Nov 03 19:46:47.437434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.437037    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="debedf10686728bc293f08e3f3a1954f8c624e213da1e3c5b61a6972b9373c1d"
Nov 03 19:46:47.437434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.436975    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"
Nov 03 19:46:47.439426 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.439383    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp" event={"ID":"5861fe0f-fe8a-4704-8a51-689479fac3c9","Type":"ContainerStarted","Data":"2a0abb67cc9fabaf06a54962c6af49dfb91f64d3ded0fdb4c21de95dd4ea2659"}
Nov 03 19:46:47.455968 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.455918    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp" podStartSLOduration=2.455904465 podStartE2EDuration="2.455904465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:47.452430463 +0000 UTC m=+920.722553758" watchObservedRunningTime="2026-11-03 19:46:47.455904465 +0000 UTC m=+920.726027760"
Nov 03 19:46:47.517006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.509442    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2ea3ef5-cc5cc\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:47.517006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.509538    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2ea3ef5-cc5cc\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:47.517006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.509732    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgh7n\" (UniqueName: \"kubernetes.io/projected/bd166e04-e728-426b-8222-f251ea015d6f-kube-api-access-lgh7n\") pod \"maas-api-revoke-sub-c2ea3ef5-cc5cc\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:47.517006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.511718    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2ea3ef5-cc5cc\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:47.523198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.522805    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgh7n\" (UniqueName: \"kubernetes.io/projected/bd166e04-e728-426b-8222-f251ea015d6f-kube-api-access-lgh7n\") pod \"maas-api-revoke-sub-c2ea3ef5-cc5cc\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:47.528555 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.528521    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2ea3ef5-cc5cc\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:47.603814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.603781    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:47.641054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.641022    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:47.712275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.712243    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-kube-api-access-9c8x5\" (UniqueName: \"kubernetes.io/projected/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-kube-api-access-9c8x5\") pod \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") "
Nov 03 19:46:47.712590 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.712421    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-xks-ca\") pod \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") "
Nov 03 19:46:47.712590 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.712504    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-openshift-ca\") pod \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\" (UID: \"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964\") "
Nov 03 19:46:47.713667 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.713613    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-openshift-ca" pod "7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964" (UID: "7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:47.715917 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.715894    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-xks-ca" pod "7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964" (UID: "7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:47.726210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.717315    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-kube-api-access-9c8x5" pod "7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964" (UID: "7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964"). InnerVolumeSpecName "kube-api-access-9c8x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:47.834905 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.834814    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:47.840122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.840029    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:47.840122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.840065    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:47.840122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.840082    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c8x5\" (UniqueName: \"kubernetes.io/projected/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964-kube-api-access-9c8x5\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:47.940455 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:47.940424    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd166e04_e728_426b_8222_f251ea015d6f.slice/crio-6f1aa3753adc4fe97acbb321b118ad064e5064431fcc9d445b274d3c3524c73a WatchSource:0}: Error finding container 6f1aa3753adc4fe97acbb321b118ad064e5064431fcc9d445b274d3c3524c73a: Status 404 returned error can't find the container with id 6f1aa3753adc4fe97acbb321b118ad064e5064431fcc9d445b274d3c3524c73a
Nov 03 19:46:47.944276 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:47.944249    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"]
Nov 03 19:46:48.449344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:48.449259    4531 generic.go:356] "Generic (PLEG): container finished" podID="5861fe0f-fe8a-4704-8a51-689479fac3c9" containerID="2a0abb67cc9fabaf06a54962c6af49dfb91f64d3ded0fdb4c21de95dd4ea2659" exitCode=0
Nov 03 19:46:48.449709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:48.449343    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp" event={"ID":"5861fe0f-fe8a-4704-8a51-689479fac3c9","Type":"ContainerDied","Data":"2a0abb67cc9fabaf06a54962c6af49dfb91f64d3ded0fdb4c21de95dd4ea2659"}
Nov 03 19:46:48.450922 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:48.450901    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"
Nov 03 19:46:48.451068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:48.450904    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp" event={"ID":"7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964","Type":"ContainerDied","Data":"77f2660f95a8a3a570621ce648d1efd2a58d978441a5cc69e601f3451abb6011"}
Nov 03 19:46:48.451068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:48.451019    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77f2660f95a8a3a570621ce648d1efd2a58d978441a5cc69e601f3451abb6011"
Nov 03 19:46:48.452956 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:48.452726    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc" event={"ID":"bd166e04-e728-426b-8222-f251ea015d6f","Type":"ContainerStarted","Data":"e8fc30bc626ec738309997f18dbc2ff1c9523f34ae648227a481ef2da6fd2ac8"}
Nov 03 19:46:48.452956 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:48.452782    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc" event={"ID":"bd166e04-e728-426b-8222-f251ea015d6f","Type":"ContainerStarted","Data":"6f1aa3753adc4fe97acbb321b118ad064e5064431fcc9d445b274d3c3524c73a"}
Nov 03 19:46:48.478676 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:48.478624    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc" podStartSLOduration=1.4786114910000001 podStartE2EDuration="1.478611491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:48.475948262 +0000 UTC m=+921.746071557" watchObservedRunningTime="2026-11-03 19:46:48.478611491 +0000 UTC m=+921.748734786"
Nov 03 19:46:48.510763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:48.510730    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:49.148995 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.148961    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:49.462836 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.462739    4531 generic.go:356] "Generic (PLEG): container finished" podID="bd166e04-e728-426b-8222-f251ea015d6f" containerID="e8fc30bc626ec738309997f18dbc2ff1c9523f34ae648227a481ef2da6fd2ac8" exitCode=0
Nov 03 19:46:49.463284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.462827    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc" event={"ID":"bd166e04-e728-426b-8222-f251ea015d6f","Type":"ContainerDied","Data":"e8fc30bc626ec738309997f18dbc2ff1c9523f34ae648227a481ef2da6fd2ac8"}
Nov 03 19:46:49.654527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.654502    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:49.780753 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.780671    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-xks-ca\") pod \"5861fe0f-fe8a-4704-8a51-689479fac3c9\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") "
Nov 03 19:46:49.780753 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.780747    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5861fe0f-fe8a-4704-8a51-689479fac3c9-kube-api-access-69tsp\" (UniqueName: \"kubernetes.io/projected/5861fe0f-fe8a-4704-8a51-689479fac3c9-kube-api-access-69tsp\") pod \"5861fe0f-fe8a-4704-8a51-689479fac3c9\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") "
Nov 03 19:46:49.781119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.781100    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-openshift-ca\") pod \"5861fe0f-fe8a-4704-8a51-689479fac3c9\" (UID: \"5861fe0f-fe8a-4704-8a51-689479fac3c9\") "
Nov 03 19:46:49.781547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.781519    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-openshift-ca" pod "5861fe0f-fe8a-4704-8a51-689479fac3c9" (UID: "5861fe0f-fe8a-4704-8a51-689479fac3c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:49.781773 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.781754    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:49.782792 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.782772    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-xks-ca" pod "5861fe0f-fe8a-4704-8a51-689479fac3c9" (UID: "5861fe0f-fe8a-4704-8a51-689479fac3c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:49.782892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.782868    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5861fe0f-fe8a-4704-8a51-689479fac3c9-kube-api-access-69tsp" pod "5861fe0f-fe8a-4704-8a51-689479fac3c9" (UID: "5861fe0f-fe8a-4704-8a51-689479fac3c9"). InnerVolumeSpecName "kube-api-access-69tsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:49.851213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.849756    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:49.885141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.885103    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5861fe0f-fe8a-4704-8a51-689479fac3c9-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:49.885141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:49.885138    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69tsp\" (UniqueName: \"kubernetes.io/projected/5861fe0f-fe8a-4704-8a51-689479fac3c9-kube-api-access-69tsp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:50.455200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.452212    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:50.522196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.520342    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"
Nov 03 19:46:50.528550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.528393    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp" event={"ID":"5861fe0f-fe8a-4704-8a51-689479fac3c9","Type":"ContainerDied","Data":"ebb1fa3a8ddbd506b121482fd03899ae2857c543013dd2278a4acfc7ee4fd5e0"}
Nov 03 19:46:50.528550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.528460    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebb1fa3a8ddbd506b121482fd03899ae2857c543013dd2278a4acfc7ee4fd5e0"
Nov 03 19:46:50.578116 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:46:50.578079    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:46:50.709553 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.709496    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:50.799645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.799614    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-openshift-ca\") pod \"bd166e04-e728-426b-8222-f251ea015d6f\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") "
Nov 03 19:46:50.799815 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.799693    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-xks-ca\") pod \"bd166e04-e728-426b-8222-f251ea015d6f\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") "
Nov 03 19:46:50.799815 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.799783    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd166e04-e728-426b-8222-f251ea015d6f-kube-api-access-lgh7n\" (UniqueName: \"kubernetes.io/projected/bd166e04-e728-426b-8222-f251ea015d6f-kube-api-access-lgh7n\") pod \"bd166e04-e728-426b-8222-f251ea015d6f\" (UID: \"bd166e04-e728-426b-8222-f251ea015d6f\") "
Nov 03 19:46:50.800028 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.800002    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-openshift-ca" pod "bd166e04-e728-426b-8222-f251ea015d6f" (UID: "bd166e04-e728-426b-8222-f251ea015d6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:50.800411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.800393    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:50.801802 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.801776    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd166e04-e728-426b-8222-f251ea015d6f-kube-api-access-lgh7n" pod "bd166e04-e728-426b-8222-f251ea015d6f" (UID: "bd166e04-e728-426b-8222-f251ea015d6f"). InnerVolumeSpecName "kube-api-access-lgh7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:50.801905 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.801859    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-xks-ca" pod "bd166e04-e728-426b-8222-f251ea015d6f" (UID: "bd166e04-e728-426b-8222-f251ea015d6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:50.901526 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.901485    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd166e04-e728-426b-8222-f251ea015d6f-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:50.901526 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:50.901520    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgh7n\" (UniqueName: \"kubernetes.io/projected/bd166e04-e728-426b-8222-f251ea015d6f-kube-api-access-lgh7n\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:51.129251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.120872    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:51.471451 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.471372    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"]
Nov 03 19:46:51.472084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.472069    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964" containerName="revoke-keys"
Nov 03 19:46:51.472130 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.472086    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964" containerName="revoke-keys"
Nov 03 19:46:51.472216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.472129    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5861fe0f-fe8a-4704-8a51-689479fac3c9" containerName="revoke-keys"
Nov 03 19:46:51.472216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.472134    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5861fe0f-fe8a-4704-8a51-689479fac3c9" containerName="revoke-keys"
Nov 03 19:46:51.472216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.472151    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bd166e04-e728-426b-8222-f251ea015d6f" containerName="revoke-keys"
Nov 03 19:46:51.472216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.472156    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd166e04-e728-426b-8222-f251ea015d6f" containerName="revoke-keys"
Nov 03 19:46:51.472325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.472270    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5861fe0f-fe8a-4704-8a51-689479fac3c9" containerName="revoke-keys"
Nov 03 19:46:51.472325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.472280    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd166e04-e728-426b-8222-f251ea015d6f" containerName="revoke-keys"
Nov 03 19:46:51.472325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.472289    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964" containerName="revoke-keys"
Nov 03 19:46:51.476336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.476309    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:51.483226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.483198    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"]
Nov 03 19:46:51.530470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.530436    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"
Nov 03 19:46:51.530470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.530458    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc" event={"ID":"bd166e04-e728-426b-8222-f251ea015d6f","Type":"ContainerDied","Data":"6f1aa3753adc4fe97acbb321b118ad064e5064431fcc9d445b274d3c3524c73a"}
Nov 03 19:46:51.530928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.530522    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f1aa3753adc4fe97acbb321b118ad064e5064431fcc9d445b274d3c3524c73a"
Nov 03 19:46:51.538663 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.538634    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742b9c55-vzqrl\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:51.538825 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.538674    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742b9c55-vzqrl\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:51.538959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.538943    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcdtr\" (UniqueName: \"kubernetes.io/projected/e5c2ec68-1457-4b91-8efb-14cf29caa92e-kube-api-access-lcdtr\") pod \"maas-api-revoke-sub-742b9c55-vzqrl\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:51.640681 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.640642    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcdtr\" (UniqueName: \"kubernetes.io/projected/e5c2ec68-1457-4b91-8efb-14cf29caa92e-kube-api-access-lcdtr\") pod \"maas-api-revoke-sub-742b9c55-vzqrl\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:51.640905 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.640767    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742b9c55-vzqrl\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:51.640905 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.640791    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742b9c55-vzqrl\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:51.641487 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.641465    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742b9c55-vzqrl\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:51.643426 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.643397    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742b9c55-vzqrl\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:51.647901 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.647880    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcdtr\" (UniqueName: \"kubernetes.io/projected/e5c2ec68-1457-4b91-8efb-14cf29caa92e-kube-api-access-lcdtr\") pod \"maas-api-revoke-sub-742b9c55-vzqrl\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:51.797434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:51.797348    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:52.006584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:52.005931    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:52.085524 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:46:52.085492    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c2ec68_1457_4b91_8efb_14cf29caa92e.slice/crio-16a3e0a09c655bda2ab75ecb3a0990a9764047bef0f43524b26855c0393cd405 WatchSource:0}: Error finding container 16a3e0a09c655bda2ab75ecb3a0990a9764047bef0f43524b26855c0393cd405: Status 404 returned error can't find the container with id 16a3e0a09c655bda2ab75ecb3a0990a9764047bef0f43524b26855c0393cd405
Nov 03 19:46:52.085524 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:52.085515    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"]
Nov 03 19:46:52.331506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:52.331469    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:52.464645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:52.464606    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:52.541133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:52.541087    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl" event={"ID":"e5c2ec68-1457-4b91-8efb-14cf29caa92e","Type":"ContainerStarted","Data":"6d7b41d9d0523457e2d7fb576944aca810853eb8385b3c798a6ebbbf0149cc68"}
Nov 03 19:46:52.541554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:52.541187    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl" event={"ID":"e5c2ec68-1457-4b91-8efb-14cf29caa92e","Type":"ContainerStarted","Data":"16a3e0a09c655bda2ab75ecb3a0990a9764047bef0f43524b26855c0393cd405"}
Nov 03 19:46:52.563909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:52.563852    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl" podStartSLOduration=1.563837653 podStartE2EDuration="1.563837653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:52.559260365 +0000 UTC m=+925.829383660" watchObservedRunningTime="2026-11-03 19:46:52.563837653 +0000 UTC m=+925.833960947"
Nov 03 19:46:52.618578 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:52.618483    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:52.770890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:52.770828    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:53.051189 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:53.051135    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:53.200368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:53.200314    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:53.454138 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:53.454039    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:53.553903 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:53.553865    4531 generic.go:356] "Generic (PLEG): container finished" podID="e5c2ec68-1457-4b91-8efb-14cf29caa92e" containerID="6d7b41d9d0523457e2d7fb576944aca810853eb8385b3c798a6ebbbf0149cc68" exitCode=0
Nov 03 19:46:53.554350 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:53.553949    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl" event={"ID":"e5c2ec68-1457-4b91-8efb-14cf29caa92e","Type":"ContainerDied","Data":"6d7b41d9d0523457e2d7fb576944aca810853eb8385b3c798a6ebbbf0149cc68"}
Nov 03 19:46:54.601953 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:54.599463    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:54.752124 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:54.750326    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:46:54.945363 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:54.901274    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:54.945363 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:54.901698    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:46:54.945363 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:54.902297    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:46:54.964197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:54.963980    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:46:54.994207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:54.964411    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:46:55.068140 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.068086    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.126230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.104594    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:46:55.154899 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.154855    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.326445 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.326339    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.374491 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.374441    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:55.426788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.426296    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.447811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.447764    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.751438 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.751390    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:46:55.752969 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.752944    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:46:55.753431 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.753383    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Nov 03 19:46:55.940683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.901932    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:46:55.940683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.902397    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.940683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.903227    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.940683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.904897    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:46:55.940683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.904957    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:46:55.940683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:55.904993    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:46:56.053455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.046451    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:56.053455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.046847    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:56.053455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.049194    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:46:56.053455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.049255    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:46:56.118208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.115128    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:56.118208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.117144    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:46:56.229216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.221134    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:46:56.291349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.291308    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:46:56.363198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.362477    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:46:56.382197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.370441    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:46:56.802532 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.475133    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:56.814286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.634746    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Nov 03 19:46:56.814286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.746625    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:46:56.814286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.806392    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:46:56.814286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.806453    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:46:56.814286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.752642    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:46:56.814286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.746663    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:46:56.822218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.814844    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:46:57.012208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:56.961253    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.084034 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.083931    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.115977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.115909    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.154416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.120800    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.154416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.121739    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.154756 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.154658    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.267244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.238152    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.373391 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.373286    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.412304 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.412257    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:57.412581 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.412549    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.526876 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.526830    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:57.527195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.527141    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.685199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.684301    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:57.685199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.684373    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.730441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.729306    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:57.730441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.729377    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.832230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.828280    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.832230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.828597    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.832230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.828792    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.832230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.828967    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.846940 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.845276    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:57.846940 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:57.845347    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.015299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.014096    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:46:58.021506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.021404    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:58.093327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.093279    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.118395 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.118346    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.171829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.171784    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.272253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.208284    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.272253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.218498    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:46:58.273962 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.273397    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:46:58.417971 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.360799    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.417971 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.387487    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.540246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.513235    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:58.847043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.846646    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.847855    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.848268    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.849906    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.850254    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.850530    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.856272    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.857486    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c2ec68-1457-4b91-8efb-14cf29caa92e-kube-api-access-lcdtr\" (UniqueName: \"kubernetes.io/projected/e5c2ec68-1457-4b91-8efb-14cf29caa92e-kube-api-access-lcdtr\") pod \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") "
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.857560    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-openshift-ca\") pod \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") "
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.857681    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-xks-ca\") pod \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\" (UID: \"e5c2ec68-1457-4b91-8efb-14cf29caa92e\") "
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.866655    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-openshift-ca" pod "e5c2ec68-1457-4b91-8efb-14cf29caa92e" (UID: "e5c2ec68-1457-4b91-8efb-14cf29caa92e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.919717    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-xks-ca" pod "e5c2ec68-1457-4b91-8efb-14cf29caa92e" (UID: "e5c2ec68-1457-4b91-8efb-14cf29caa92e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.932758    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c2ec68-1457-4b91-8efb-14cf29caa92e-kube-api-access-lcdtr" pod "e5c2ec68-1457-4b91-8efb-14cf29caa92e" (UID: "e5c2ec68-1457-4b91-8efb-14cf29caa92e"). InnerVolumeSpecName "kube-api-access-lcdtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.960813    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcdtr\" (UniqueName: \"kubernetes.io/projected/e5c2ec68-1457-4b91-8efb-14cf29caa92e-kube-api-access-lcdtr\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.960872    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:58.960910    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c2ec68-1457-4b91-8efb-14cf29caa92e-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:59.043481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.018268    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.121433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.121320    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.334198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.148782    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.349206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.221267    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.389665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.364062    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.492642 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.221304    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.492642 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.463427    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.492642 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.221330    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.494263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.223993    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:59.571298 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.571256    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.858261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.856282    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.858261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.857569    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.858811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.858745    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.859928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.859869    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.895347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.895283    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl" event={"ID":"e5c2ec68-1457-4b91-8efb-14cf29caa92e","Type":"ContainerDied","Data":"16a3e0a09c655bda2ab75ecb3a0990a9764047bef0f43524b26855c0393cd405"}
Nov 03 19:46:59.895995 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.895951    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16a3e0a09c655bda2ab75ecb3a0990a9764047bef0f43524b26855c0393cd405"
Nov 03 19:46:59.896647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:46:59.896536    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"
Nov 03 19:47:00.034256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.020328    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.336203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.335274    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.336203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.335750    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:47:00.337500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.337464    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:47:00.366850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.366807    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"]
Nov 03 19:47:00.372263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.372227    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e5c2ec68-1457-4b91-8efb-14cf29caa92e" containerName="revoke-keys"
Nov 03 19:47:00.372476 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.372457    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c2ec68-1457-4b91-8efb-14cf29caa92e" containerName="revoke-keys"
Nov 03 19:47:00.372849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.372828    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c2ec68-1457-4b91-8efb-14cf29caa92e" containerName="revoke-keys"
Nov 03 19:47:00.385545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.385510    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.406207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.400153    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:47:00.406207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.402578    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2e17271b-openshift-default-dockercfg-pdbqv\""
Nov 03 19:47:00.440381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.440339    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:00.440658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.440624    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.453552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.453243    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"]
Nov 03 19:47:00.458302 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.458257    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.475360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.475321    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-podinfo\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.475652 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.475625    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-certs\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.479198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.475851    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dnsl\" (UniqueName: \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-kube-api-access-8dnsl\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.498198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.479551    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istiod-ca-cert\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.498454 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.475424    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.498688 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.498651    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-data\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.498791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.498707    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-token\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.498791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.498754    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.498889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.498816    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-envoy\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.498956 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.498938    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-credential-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.499351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.475469    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.604515 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.604370    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-podinfo\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.637626 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.637587    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-certs\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.637946 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.637923    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dnsl\" (UniqueName: \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-kube-api-access-8dnsl\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.638269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.638247    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istiod-ca-cert\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.638499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.638481    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-data\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.638618 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.638603    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-token\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.638742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.638727    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.638897 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.638880    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-envoy\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.643279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.639191    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-credential-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.643593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.641995    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istiod-ca-cert\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.643725 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.636936    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-podinfo\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.643835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.642317    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-certs\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.643936 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.643143    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-data\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.675288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.647492    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.675288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.654386    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-credential-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.693647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.693530    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-envoy\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.694537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.693838    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dnsl\" (UniqueName: \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-kube-api-access-8dnsl\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.698884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.698852    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-token\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:00.881277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.865516    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.881277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.865997    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:47:00.881277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.866360    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:47:00.881277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.866623    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:47:00.882782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:00.882745    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:01.049212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.041675    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:47:01.170114 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.169923    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:01.343212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.336263    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:47:01.343212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.336738    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:47:01.343212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.338152    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:47:01.401846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.401794    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.429882 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.429792    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:01.485334 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.485281    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.501682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.501636    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.508547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.507309    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:47:01.655319 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.655265    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.874273 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.871309    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.874273 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.873424    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:01.874273 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.873812    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:47:01.874273 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:01.874107    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.059206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.058188    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.392387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.392336    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.422200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.394538    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.696678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.413642    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.696678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.431231    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.696678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.636698    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.696678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.636736    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.696678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.652653    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.921258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.871923    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.921258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.876076    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.921258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.876712    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.921258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.877279    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.965199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.961769    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:02.965199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.961873    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:02.965199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.961912    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:02.978455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:02.978427    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:47:03.098012 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.067562    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.466375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.450252    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:47:03.480102 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.471903    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:47:03.510280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.509330    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 03 19:47:03.742186 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.739449    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:47:03.742186 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.739693    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:47:03.815704 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.815330    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:47:03.828238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.815962    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:47:03.897567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.888556    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.897567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.891368    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:47:03.897567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.891596    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:47:03.918242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.918205    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.975940 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.961157    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"]
Nov 03 19:47:03.975940 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.961296    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:47:03.975940 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.967923    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:47:03.975940 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.974529    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:47:03.982542 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.981136    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:47:03.988056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.987690    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:47:03.988056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.987764    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr" event={"ID":"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8","Type":"ContainerStarted","Data":"61d5f1e89d1b1964980f1155104d65cb8f1cff70f32b2031a81cbd96a51f1ac8"}
Nov 03 19:47:03.988311 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.988274    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:47:03.988375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.988340    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:47:03.994603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:03.994543    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:47:04.021548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.021493    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"]
Nov 03 19:47:04.066381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.061518    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:47:04.066381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.061558    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"]
Nov 03 19:47:04.066381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.061607    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:47:04.066381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.062651    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:04.071228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.070944    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:47:04.071592 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.071576    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:47:04.113970 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.113943    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"]
Nov 03 19:47:04.120547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.120516    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:04.121286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.121255    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f16ed89-89jhh\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:04.122747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.121454    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5gv\" (UniqueName: \"kubernetes.io/projected/361de3fa-82d0-408e-b46c-b29a66a68c60-kube-api-access-sq5gv\") pod \"maas-api-revoke-sub-7f16ed89-89jhh\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:04.123646 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.122942    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"]
Nov 03 19:47:04.123867 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.123744    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f16ed89-89jhh\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:04.225653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.225611    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40eec696-fvsvn\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:04.225885 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.225824    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f16ed89-89jhh\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:04.225885 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.225856    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklqw\" (UniqueName: \"kubernetes.io/projected/0120cf8e-8eca-48a0-918a-c13fc4baeb05-kube-api-access-fklqw\") pod \"maas-api-revoke-sub-40eec696-fvsvn\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:04.226060 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.226037    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f16ed89-89jhh\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:04.226282 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.226265    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40eec696-fvsvn\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:04.226360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.226300    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5gv\" (UniqueName: \"kubernetes.io/projected/361de3fa-82d0-408e-b46c-b29a66a68c60-kube-api-access-sq5gv\") pod \"maas-api-revoke-sub-7f16ed89-89jhh\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:04.227271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.227247    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f16ed89-89jhh\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:04.228741 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.228716    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f16ed89-89jhh\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:04.236947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.236919    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5gv\" (UniqueName: \"kubernetes.io/projected/361de3fa-82d0-408e-b46c-b29a66a68c60-kube-api-access-sq5gv\") pod \"maas-api-revoke-sub-7f16ed89-89jhh\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:04.327992 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.327906    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40eec696-fvsvn\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:04.328168 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.328095    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40eec696-fvsvn\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:04.328916 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.328890    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40eec696-fvsvn\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:04.329055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.329038    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklqw\" (UniqueName: \"kubernetes.io/projected/0120cf8e-8eca-48a0-918a-c13fc4baeb05-kube-api-access-fklqw\") pod \"maas-api-revoke-sub-40eec696-fvsvn\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:04.331423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.331396    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40eec696-fvsvn\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:04.345957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.345929    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklqw\" (UniqueName: \"kubernetes.io/projected/0120cf8e-8eca-48a0-918a-c13fc4baeb05-kube-api-access-fklqw\") pod \"maas-api-revoke-sub-40eec696-fvsvn\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:04.388095 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.388059    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:04.479929 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.479894    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:04.697502 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:04.697470    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361de3fa_82d0_408e_b46c_b29a66a68c60.slice/crio-98baedca2857d96da04929947e6f846cff11b77f508353615648c79f7fad5715 WatchSource:0}: Error finding container 98baedca2857d96da04929947e6f846cff11b77f508353615648c79f7fad5715: Status 404 returned error can't find the container with id 98baedca2857d96da04929947e6f846cff11b77f508353615648c79f7fad5715
Nov 03 19:47:04.698163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.698135    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"]
Nov 03 19:47:04.797124 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.797090    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"]
Nov 03 19:47:04.797929 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:04.797898    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0120cf8e_8eca_48a0_918a_c13fc4baeb05.slice/crio-6915789b53a0ae13847bb8c869a3d6dcda8f88db18f61354b7bb0b6560a00954 WatchSource:0}: Error finding container 6915789b53a0ae13847bb8c869a3d6dcda8f88db18f61354b7bb0b6560a00954: Status 404 returned error can't find the container with id 6915789b53a0ae13847bb8c869a3d6dcda8f88db18f61354b7bb0b6560a00954
Nov 03 19:47:04.892068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.892036    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:04.955066 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.955030    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"]
Nov 03 19:47:04.960947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.960921    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:04.986528 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.986375    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cbfcf47a\""
Nov 03 19:47:04.986528 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.986431    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cbfcf47a-dockercfg-6pv68\""
Nov 03 19:47:04.998547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:04.998517    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"]
Nov 03 19:47:05.004094 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.004052    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr" event={"ID":"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8","Type":"ContainerStarted","Data":"18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719"}
Nov 03 19:47:05.008567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.008536    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh" event={"ID":"361de3fa-82d0-408e-b46c-b29a66a68c60","Type":"ContainerStarted","Data":"98baedca2857d96da04929947e6f846cff11b77f508353615648c79f7fad5715"}
Nov 03 19:47:05.010812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.010782    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn" event={"ID":"0120cf8e-8eca-48a0-918a-c13fc4baeb05","Type":"ContainerStarted","Data":"6915789b53a0ae13847bb8c869a3d6dcda8f88db18f61354b7bb0b6560a00954"}
Nov 03 19:47:05.039711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.039674    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91501c4c-a6cc-479b-b660-17a0604c7346-plugins-config-volume\") pod \"payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv\" (UID: \"91501c4c-a6cc-479b-b660-17a0604c7346\") " pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:05.039910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.039891    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24642\" (UniqueName: \"kubernetes.io/projected/91501c4c-a6cc-479b-b660-17a0604c7346-kube-api-access-24642\") pod \"payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv\" (UID: \"91501c4c-a6cc-479b-b660-17a0604c7346\") " pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:05.042330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.042282    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr" podStartSLOduration=7.042264159 podStartE2EDuration="7.042264159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:05.03466757 +0000 UTC m=+938.304790869" watchObservedRunningTime="2026-11-03 19:47:05.042264159 +0000 UTC m=+938.312387457"
Nov 03 19:47:05.099062 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.098985    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:05.141821 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.141786    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24642\" (UniqueName: \"kubernetes.io/projected/91501c4c-a6cc-479b-b660-17a0604c7346-kube-api-access-24642\") pod \"payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv\" (UID: \"91501c4c-a6cc-479b-b660-17a0604c7346\") " pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:05.142258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.141994    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91501c4c-a6cc-479b-b660-17a0604c7346-plugins-config-volume\") pod \"payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv\" (UID: \"91501c4c-a6cc-479b-b660-17a0604c7346\") " pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:05.142723 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.142702    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91501c4c-a6cc-479b-b660-17a0604c7346-plugins-config-volume\") pod \"payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv\" (UID: \"91501c4c-a6cc-479b-b660-17a0604c7346\") " pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:05.149001 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.148972    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24642\" (UniqueName: \"kubernetes.io/projected/91501c4c-a6cc-479b-b660-17a0604c7346-kube-api-access-24642\") pod \"payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv\" (UID: \"91501c4c-a6cc-479b-b660-17a0604c7346\") " pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:05.283939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.283907    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:05.795454 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:05.795416    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91501c4c_a6cc_479b_b660_17a0604c7346.slice/crio-ffb9074fcf4ed0d20091eec0c56e86e6766c2542f1a3e1cd506405fea416edb5 WatchSource:0}: Error finding container ffb9074fcf4ed0d20091eec0c56e86e6766c2542f1a3e1cd506405fea416edb5: Status 404 returned error can't find the container with id ffb9074fcf4ed0d20091eec0c56e86e6766c2542f1a3e1cd506405fea416edb5
Nov 03 19:47:05.797386 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.797359    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"]
Nov 03 19:47:05.883926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.883899    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:05.890862 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:05.890834    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:06.021791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.021755    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh" event={"ID":"361de3fa-82d0-408e-b46c-b29a66a68c60","Type":"ContainerStarted","Data":"f8786709590d5108fb93cb8e810f40c82bf288ef063a5b747e4a92a8c33512f8"}
Nov 03 19:47:06.023810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.023777    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv" event={"ID":"91501c4c-a6cc-479b-b660-17a0604c7346","Type":"ContainerStarted","Data":"bdb46d723768ca21340767fcbf00661e9122e8f9cb2d65726b0b042853dac682"}
Nov 03 19:47:06.023979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.023834    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:06.023979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.023854    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv" event={"ID":"91501c4c-a6cc-479b-b660-17a0604c7346","Type":"ContainerStarted","Data":"ffb9074fcf4ed0d20091eec0c56e86e6766c2542f1a3e1cd506405fea416edb5"}
Nov 03 19:47:06.025734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.025709    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn" event={"ID":"0120cf8e-8eca-48a0-918a-c13fc4baeb05","Type":"ContainerStarted","Data":"5a612058689b69f6b24376441ef0ea9c4091a08f7ee202c471b93e95b55e1010"}
Nov 03 19:47:06.025913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.025895    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:06.026907 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.026884    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:06.042233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.042109    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh" podStartSLOduration=3.042078747 podStartE2EDuration="3.042078747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:06.035009061 +0000 UTC m=+939.305132355" watchObservedRunningTime="2026-11-03 19:47:06.042078747 +0000 UTC m=+939.312202040"
Nov 03 19:47:06.078692 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.078623    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn" podStartSLOduration=2.078603974 podStartE2EDuration="2.078603974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:06.067940181 +0000 UTC m=+939.338063477" watchObservedRunningTime="2026-11-03 19:47:06.078603974 +0000 UTC m=+939.348727270"
Nov 03 19:47:06.093193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.090822    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv" podStartSLOduration=2.090804567 podStartE2EDuration="2.090804567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:06.08202108 +0000 UTC m=+939.352144386" watchObservedRunningTime="2026-11-03 19:47:06.090804567 +0000 UTC m=+939.360927862"
Nov 03 19:47:06.230781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.230741    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:06.600223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.600190    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:06.747331 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:06.747280    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:07.037289 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.037254    4531 generic.go:356] "Generic (PLEG): container finished" podID="361de3fa-82d0-408e-b46c-b29a66a68c60" containerID="f8786709590d5108fb93cb8e810f40c82bf288ef063a5b747e4a92a8c33512f8" exitCode=0
Nov 03 19:47:07.037444 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.037337    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh" event={"ID":"361de3fa-82d0-408e-b46c-b29a66a68c60","Type":"ContainerDied","Data":"f8786709590d5108fb93cb8e810f40c82bf288ef063a5b747e4a92a8c33512f8"}
Nov 03 19:47:07.039387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.039355    4531 generic.go:356] "Generic (PLEG): container finished" podID="0120cf8e-8eca-48a0-918a-c13fc4baeb05" containerID="5a612058689b69f6b24376441ef0ea9c4091a08f7ee202c471b93e95b55e1010" exitCode=0
Nov 03 19:47:07.039578 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.039432    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn" event={"ID":"0120cf8e-8eca-48a0-918a-c13fc4baeb05","Type":"ContainerDied","Data":"5a612058689b69f6b24376441ef0ea9c4091a08f7ee202c471b93e95b55e1010"}
Nov 03 19:47:07.143262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.143227    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:07.293260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.290694    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:07.672871 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.672833    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:07.820296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.820260    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:07.877053 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.877021    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"]
Nov 03 19:47:07.885455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.885427    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:07.889738 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.889706    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"]
Nov 03 19:47:07.890335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.890310    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2e17271b-dockercfg-fwd8d\""
Nov 03 19:47:07.890441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.890361    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2e17271b\""
Nov 03 19:47:07.918877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.918838    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k6rm\" (UniqueName: \"kubernetes.io/projected/93adf525-5fd1-4beb-bc15-990bbf213b5c-kube-api-access-7k6rm\") pod \"payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz\" (UID: \"93adf525-5fd1-4beb-bc15-990bbf213b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:07.919010 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.918983    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93adf525-5fd1-4beb-bc15-990bbf213b5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz\" (UID: \"93adf525-5fd1-4beb-bc15-990bbf213b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:07.935086 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.935014    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"]
Nov 03 19:47:07.943414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.943388    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:07.946505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.946481    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"]
Nov 03 19:47:07.994351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:07.966397    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:08.044194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.043419    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93adf525-5fd1-4beb-bc15-990bbf213b5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz\" (UID: \"93adf525-5fd1-4beb-bc15-990bbf213b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:08.044194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.043611    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhv9b\" (UniqueName: \"kubernetes.io/projected/38448d63-7dad-48b1-8373-b72e8ede03af-kube-api-access-rhv9b\") pod \"payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml\" (UID: \"38448d63-7dad-48b1-8373-b72e8ede03af\") " pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:08.044194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.043824    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38448d63-7dad-48b1-8373-b72e8ede03af-plugins-config-volume\") pod \"payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml\" (UID: \"38448d63-7dad-48b1-8373-b72e8ede03af\") " pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:08.044194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.044028    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k6rm\" (UniqueName: \"kubernetes.io/projected/93adf525-5fd1-4beb-bc15-990bbf213b5c-kube-api-access-7k6rm\") pod \"payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz\" (UID: \"93adf525-5fd1-4beb-bc15-990bbf213b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:08.067328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.045354    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93adf525-5fd1-4beb-bc15-990bbf213b5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz\" (UID: \"93adf525-5fd1-4beb-bc15-990bbf213b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:08.100224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.097968    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k6rm\" (UniqueName: \"kubernetes.io/projected/93adf525-5fd1-4beb-bc15-990bbf213b5c-kube-api-access-7k6rm\") pod \"payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz\" (UID: \"93adf525-5fd1-4beb-bc15-990bbf213b5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:08.173202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.149480    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhv9b\" (UniqueName: \"kubernetes.io/projected/38448d63-7dad-48b1-8373-b72e8ede03af-kube-api-access-rhv9b\") pod \"payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml\" (UID: \"38448d63-7dad-48b1-8373-b72e8ede03af\") " pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:08.173202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.149647    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38448d63-7dad-48b1-8373-b72e8ede03af-plugins-config-volume\") pod \"payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml\" (UID: \"38448d63-7dad-48b1-8373-b72e8ede03af\") " pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:08.191246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.176608    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38448d63-7dad-48b1-8373-b72e8ede03af-plugins-config-volume\") pod \"payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml\" (UID: \"38448d63-7dad-48b1-8373-b72e8ede03af\") " pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:08.256193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.229371    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:08.311355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.310719    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:08.433198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.430549    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhv9b\" (UniqueName: \"kubernetes.io/projected/38448d63-7dad-48b1-8373-b72e8ede03af-kube-api-access-rhv9b\") pod \"payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml\" (UID: \"38448d63-7dad-48b1-8373-b72e8ede03af\") " pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:08.589838 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.576902    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:08.773357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.772739    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:08.778048 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.778005    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"]
Nov 03 19:47:08.806012 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.805936    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-xks-ca\") pod \"361de3fa-82d0-408e-b46c-b29a66a68c60\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") "
Nov 03 19:47:08.807694 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.806395    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361de3fa-82d0-408e-b46c-b29a66a68c60-kube-api-access-sq5gv\" (UniqueName: \"kubernetes.io/projected/361de3fa-82d0-408e-b46c-b29a66a68c60-kube-api-access-sq5gv\") pod \"361de3fa-82d0-408e-b46c-b29a66a68c60\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") "
Nov 03 19:47:08.807694 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.806497    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-openshift-ca\") pod \"361de3fa-82d0-408e-b46c-b29a66a68c60\" (UID: \"361de3fa-82d0-408e-b46c-b29a66a68c60\") "
Nov 03 19:47:08.807886 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.807825    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-openshift-ca" pod "361de3fa-82d0-408e-b46c-b29a66a68c60" (UID: "361de3fa-82d0-408e-b46c-b29a66a68c60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:08.809943 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.809914    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-xks-ca" pod "361de3fa-82d0-408e-b46c-b29a66a68c60" (UID: "361de3fa-82d0-408e-b46c-b29a66a68c60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:08.811262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.811235    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361de3fa-82d0-408e-b46c-b29a66a68c60-kube-api-access-sq5gv" pod "361de3fa-82d0-408e-b46c-b29a66a68c60" (UID: "361de3fa-82d0-408e-b46c-b29a66a68c60"). InnerVolumeSpecName "kube-api-access-sq5gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:08.882364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.882338    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:08.910608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.909316    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0120cf8e-8eca-48a0-918a-c13fc4baeb05-kube-api-access-fklqw\" (UniqueName: \"kubernetes.io/projected/0120cf8e-8eca-48a0-918a-c13fc4baeb05-kube-api-access-fklqw\") pod \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") "
Nov 03 19:47:08.910608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.909545    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-openshift-ca\") pod \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") "
Nov 03 19:47:08.910608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.909759    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-xks-ca\") pod \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\" (UID: \"0120cf8e-8eca-48a0-918a-c13fc4baeb05\") "
Nov 03 19:47:08.910858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.910791    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:08.910858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.910815    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq5gv\" (UniqueName: \"kubernetes.io/projected/361de3fa-82d0-408e-b46c-b29a66a68c60-kube-api-access-sq5gv\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:08.910858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.910832    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/361de3fa-82d0-408e-b46c-b29a66a68c60-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:08.914979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.914334    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-openshift-ca" pod "0120cf8e-8eca-48a0-918a-c13fc4baeb05" (UID: "0120cf8e-8eca-48a0-918a-c13fc4baeb05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:08.914979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.914553    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0120cf8e-8eca-48a0-918a-c13fc4baeb05-kube-api-access-fklqw" pod "0120cf8e-8eca-48a0-918a-c13fc4baeb05" (UID: "0120cf8e-8eca-48a0-918a-c13fc4baeb05"). InnerVolumeSpecName "kube-api-access-fklqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:08.917812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.917652    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-xks-ca" pod "0120cf8e-8eca-48a0-918a-c13fc4baeb05" (UID: "0120cf8e-8eca-48a0-918a-c13fc4baeb05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:08.917938 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:08.917910    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38448d63_7dad_48b1_8373_b72e8ede03af.slice/crio-5a8a2ab7f8dc9360af7b518c4b698254a00cda873803df32cff94c708e52099f WatchSource:0}: Error finding container 5a8a2ab7f8dc9360af7b518c4b698254a00cda873803df32cff94c708e52099f: Status 404 returned error can't find the container with id 5a8a2ab7f8dc9360af7b518c4b698254a00cda873803df32cff94c708e52099f
Nov 03 19:47:08.922964 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:08.922944    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"]
Nov 03 19:47:09.012716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.012690    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:09.012716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.012714    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0120cf8e-8eca-48a0-918a-c13fc4baeb05-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:09.012870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.012725    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fklqw\" (UniqueName: \"kubernetes.io/projected/0120cf8e-8eca-48a0-918a-c13fc4baeb05-kube-api-access-fklqw\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:09.159552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.159520    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:09.363458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.363374    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml" event={"ID":"38448d63-7dad-48b1-8373-b72e8ede03af","Type":"ContainerStarted","Data":"d931dcf3840fe03d5a9acdd6f7dd5354116f371dbe4619ccad22ecc6ee940cc5"}
Nov 03 19:47:09.363458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.363434    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml" event={"ID":"38448d63-7dad-48b1-8373-b72e8ede03af","Type":"ContainerStarted","Data":"5a8a2ab7f8dc9360af7b518c4b698254a00cda873803df32cff94c708e52099f"}
Nov 03 19:47:09.364066 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.363475    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:09.364933 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.364917    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"
Nov 03 19:47:09.365043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.364958    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh" event={"ID":"361de3fa-82d0-408e-b46c-b29a66a68c60","Type":"ContainerDied","Data":"98baedca2857d96da04929947e6f846cff11b77f508353615648c79f7fad5715"}
Nov 03 19:47:09.365043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.364998    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98baedca2857d96da04929947e6f846cff11b77f508353615648c79f7fad5715"
Nov 03 19:47:09.366866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.366841    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz" event={"ID":"93adf525-5fd1-4beb-bc15-990bbf213b5c","Type":"ContainerStarted","Data":"f37f791d03b5658e560e068e4dd3f8b70d7ae141f7540a0aa9f0993055be5869"}
Nov 03 19:47:09.366980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.366890    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz" event={"ID":"93adf525-5fd1-4beb-bc15-990bbf213b5c","Type":"ContainerStarted","Data":"279c02ad9031472d571d7ee0e224088946549bfff4ba8ddc8aaf776c92daeb90"}
Nov 03 19:47:09.366980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.366921    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:09.368832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.368813    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"
Nov 03 19:47:09.368832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.368823    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn" event={"ID":"0120cf8e-8eca-48a0-918a-c13fc4baeb05","Type":"ContainerDied","Data":"6915789b53a0ae13847bb8c869a3d6dcda8f88db18f61354b7bb0b6560a00954"}
Nov 03 19:47:09.369106 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.368862    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6915789b53a0ae13847bb8c869a3d6dcda8f88db18f61354b7bb0b6560a00954"
Nov 03 19:47:09.384992 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.384953    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml" podStartSLOduration=2.384940578 podStartE2EDuration="2.384940578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:09.378360213 +0000 UTC m=+942.648483510" watchObservedRunningTime="2026-11-03 19:47:09.384940578 +0000 UTC m=+942.655063898"
Nov 03 19:47:09.396469 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:09.396427    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz" podStartSLOduration=2.39641674 podStartE2EDuration="2.39641674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:09.392756788 +0000 UTC m=+942.662880082" watchObservedRunningTime="2026-11-03 19:47:09.39641674 +0000 UTC m=+942.666540096"
Nov 03 19:47:10.016535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:10.016497    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:10.758040 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:10.758003    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:10.905016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:10.904986    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:11.055455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:11.055366    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:11.601193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:11.597283    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:12.132286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.130378    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:12.299198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.297144    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:12.460248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.458633    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:12.616614 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.616585    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"]
Nov 03 19:47:12.617337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.617322    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0120cf8e-8eca-48a0-918a-c13fc4baeb05" containerName="revoke-keys"
Nov 03 19:47:12.617401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.617338    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0120cf8e-8eca-48a0-918a-c13fc4baeb05" containerName="revoke-keys"
Nov 03 19:47:12.617401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.617362    4531 cpu_manager.go:414] "containerMap: removing container" podUID="361de3fa-82d0-408e-b46c-b29a66a68c60" containerName="revoke-keys"
Nov 03 19:47:12.617401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.617367    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361de3fa-82d0-408e-b46c-b29a66a68c60" containerName="revoke-keys"
Nov 03 19:47:12.617491 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.617478    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="361de3fa-82d0-408e-b46c-b29a66a68c60" containerName="revoke-keys"
Nov 03 19:47:12.617523 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.617494    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0120cf8e-8eca-48a0-918a-c13fc4baeb05" containerName="revoke-keys"
Nov 03 19:47:12.622116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.622074    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.638363 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.638335    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"]
Nov 03 19:47:12.709152 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.709116    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:12.711739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.711693    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cbfcf47a-serving-cert\""
Nov 03 19:47:12.746232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.746196    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/365b7002-7797-48e2-b7bd-d8ecb75fb969-openshift-service-ca\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.746479 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.746460    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8wp9\" (UniqueName: \"kubernetes.io/projected/365b7002-7797-48e2-b7bd-d8ecb75fb969-kube-api-access-d8wp9\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.746631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.746615    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-metrics-certs\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.746671 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.746659    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-maas-api-tls\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.860202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.859200    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wp9\" (UniqueName: \"kubernetes.io/projected/365b7002-7797-48e2-b7bd-d8ecb75fb969-kube-api-access-d8wp9\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.860202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.859478    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-metrics-certs\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.860202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.859541    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-maas-api-tls\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.860202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.859717    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/365b7002-7797-48e2-b7bd-d8ecb75fb969-openshift-service-ca\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.886205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.880929    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/365b7002-7797-48e2-b7bd-d8ecb75fb969-openshift-service-ca\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.933196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.930853    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-maas-api-tls\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.933196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.931491    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-metrics-certs\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.960987 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.960941    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wp9\" (UniqueName: \"kubernetes.io/projected/365b7002-7797-48e2-b7bd-d8ecb75fb969-kube-api-access-d8wp9\") pod \"maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:12.975286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:12.975208    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:13.242730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:13.242647    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:13.321219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:13.321114    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:13.520261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:13.519107    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:13.757560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:13.757531    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:47:13.757717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:13.757696    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"]
Nov 03 19:47:13.970296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:13.970256    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:14.724574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:14.724542    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf" event={"ID":"365b7002-7797-48e2-b7bd-d8ecb75fb969","Type":"ContainerStarted","Data":"82426fd76d62e1f8bc3a3bffb9dc23c894a87ad942855bcfed0502dd7a875ecf"}
Nov 03 19:47:14.724984 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:14.724630    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:14.724984 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:14.724648    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf" event={"ID":"365b7002-7797-48e2-b7bd-d8ecb75fb969","Type":"ContainerStarted","Data":"c8bb34899451a19495e9b72b9087e0bd57b798649e9325009fc2fbcea674771a"}
Nov 03 19:47:14.743509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:14.743459    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf" podStartSLOduration=2.112797544 podStartE2EDuration="2.743443594s" totalImagesPullingTime="630.64605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:14.741233506 +0000 UTC m=+948.011356802" watchObservedRunningTime="2026-11-03 19:47:14.743443594 +0000 UTC m=+948.013566913"
Nov 03 19:47:14.878353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:14.878322    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:15.015539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.015425    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:15.137759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.137730    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:15.262023 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.261992    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"]
Nov 03 19:47:15.267019 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.266932    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:15.269854 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.269826    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:47:15.270608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.270589    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:47:15.273711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.273686    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"]
Nov 03 19:47:15.321441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.321405    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aec8afb-wnh9d\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:15.321617 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.321579    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj8b9\" (UniqueName: \"kubernetes.io/projected/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-kube-api-access-bj8b9\") pod \"maas-api-revoke-sub-3aec8afb-wnh9d\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:15.321617 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.321611    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aec8afb-wnh9d\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:15.414290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.413506    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:15.425594 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.425540    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aec8afb-wnh9d\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:15.426117 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.426074    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8b9\" (UniqueName: \"kubernetes.io/projected/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-kube-api-access-bj8b9\") pod \"maas-api-revoke-sub-3aec8afb-wnh9d\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:15.426294 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.426274    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aec8afb-wnh9d\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:15.438899 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.438867    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aec8afb-wnh9d\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:15.472596 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.465676    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj8b9\" (UniqueName: \"kubernetes.io/projected/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-kube-api-access-bj8b9\") pod \"maas-api-revoke-sub-3aec8afb-wnh9d\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:15.506120 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.478947    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aec8afb-wnh9d\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:15.588050 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.587963    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:15.888203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.881807    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:15.935900 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:15.935865    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a33b82_5427_41cb_9cd9_f2413e4e4dfb.slice/crio-42bda34cd039bdd139f73c3be4deba2c376fb64658e82189ee7db07802b89ffd WatchSource:0}: Error finding container 42bda34cd039bdd139f73c3be4deba2c376fb64658e82189ee7db07802b89ffd: Status 404 returned error can't find the container with id 42bda34cd039bdd139f73c3be4deba2c376fb64658e82189ee7db07802b89ffd
Nov 03 19:47:15.937588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:15.937558    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"]
Nov 03 19:47:16.694505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:16.694414    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:17.058819 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.054296    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:17.095410 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.095378    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"]
Nov 03 19:47:17.100289 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.100258    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d" event={"ID":"09a33b82-5427-41cb-9cd9-f2413e4e4dfb","Type":"ContainerStarted","Data":"79b0f17a6e486ca59be232dfa73ee4f8d950772f18d7f710e6d6c10b8ea0b35d"}
Nov 03 19:47:17.100449 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.100298    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d" event={"ID":"09a33b82-5427-41cb-9cd9-f2413e4e4dfb","Type":"ContainerStarted","Data":"42bda34cd039bdd139f73c3be4deba2c376fb64658e82189ee7db07802b89ffd"}
Nov 03 19:47:17.100449 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.100379    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:17.108559 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.108532    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"]
Nov 03 19:47:17.109682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.109656    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:17.117913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.117867    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d" podStartSLOduration=2.117854325 podStartE2EDuration="2.117854325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:17.116130925 +0000 UTC m=+950.386254220" watchObservedRunningTime="2026-11-03 19:47:17.117854325 +0000 UTC m=+950.387977618"
Nov 03 19:47:17.198223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.198184    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:17.280038 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.280001    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53cac197-kjgb6\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:17.280281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.280088    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvsg\" (UniqueName: \"kubernetes.io/projected/30ca9921-ca34-465e-8f96-cd15ad34e163-kube-api-access-8gvsg\") pod \"maas-api-revoke-sub-53cac197-kjgb6\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:17.280355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.280312    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53cac197-kjgb6\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:17.382278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.382245    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53cac197-kjgb6\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:17.382474 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.382295    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvsg\" (UniqueName: \"kubernetes.io/projected/30ca9921-ca34-465e-8f96-cd15ad34e163-kube-api-access-8gvsg\") pod \"maas-api-revoke-sub-53cac197-kjgb6\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:17.382474 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.382373    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53cac197-kjgb6\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:17.383004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.382981    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53cac197-kjgb6\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:17.384768 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.384750    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53cac197-kjgb6\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:17.397328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.397297    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvsg\" (UniqueName: \"kubernetes.io/projected/30ca9921-ca34-465e-8f96-cd15ad34e163-kube-api-access-8gvsg\") pod \"maas-api-revoke-sub-53cac197-kjgb6\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:17.420850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.420823    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:17.708711 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:17.708672    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ca9921_ca34_465e_8f96_cd15ad34e163.slice/crio-367111b51e627d06fc577760daa1c0c58dff251fa0b8b63f31314f990f222e94 WatchSource:0}: Error finding container 367111b51e627d06fc577760daa1c0c58dff251fa0b8b63f31314f990f222e94: Status 404 returned error can't find the container with id 367111b51e627d06fc577760daa1c0c58dff251fa0b8b63f31314f990f222e94
Nov 03 19:47:17.709901 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.709876    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"]
Nov 03 19:47:17.745453 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.745335    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:17.943678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.943647    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"]
Nov 03 19:47:17.950539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.950511    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:17.957377 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:17.957306    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"]
Nov 03 19:47:18.095781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.095749    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.096235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.095899    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c80951b-0a70-422b-acab-00b767b5304f-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.096235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.095962    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.096235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.096020    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.096235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.096068    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c80951b-0a70-422b-acab-00b767b5304f-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.096235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.096165    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4znp\" (UniqueName: \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-kube-api-access-b4znp\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.096436 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.096308    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.096436 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.096416    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.096516 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.096449    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.103094 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.103069    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6" event={"ID":"30ca9921-ca34-465e-8f96-cd15ad34e163","Type":"ContainerStarted","Data":"367111b51e627d06fc577760daa1c0c58dff251fa0b8b63f31314f990f222e94"}
Nov 03 19:47:18.104727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.104703    4531 generic.go:356] "Generic (PLEG): container finished" podID="09a33b82-5427-41cb-9cd9-f2413e4e4dfb" containerID="79b0f17a6e486ca59be232dfa73ee4f8d950772f18d7f710e6d6c10b8ea0b35d" exitCode=0
Nov 03 19:47:18.104846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.104782    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d" event={"ID":"09a33b82-5427-41cb-9cd9-f2413e4e4dfb","Type":"ContainerDied","Data":"79b0f17a6e486ca59be232dfa73ee4f8d950772f18d7f710e6d6c10b8ea0b35d"}
Nov 03 19:47:18.197551 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.197516    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.197735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.197577    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.197735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.197662    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.197856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.197763    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c80951b-0a70-422b-acab-00b767b5304f-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.197856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.197796    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.197856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.197831    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.198006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.197898    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c80951b-0a70-422b-acab-00b767b5304f-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.198006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.197964    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4znp\" (UniqueName: \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-kube-api-access-b4znp\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.198132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.198018    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.198132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.198078    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.198256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.198132    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.198256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.198185    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.198511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.198487    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.198776 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.198757    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c80951b-0a70-422b-acab-00b767b5304f-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.200038 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.200010    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.200376 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.200357    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c80951b-0a70-422b-acab-00b767b5304f-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.205270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.205225    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.205910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.205502    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4znp\" (UniqueName: \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-kube-api-access-b4znp\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.273061 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.272982    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:18.528295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.527091    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:18.578216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.578063    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"]
Nov 03 19:47:18.578333 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:18.578167    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c80951b_0a70_422b_acab_00b767b5304f.slice/crio-e8f742231d1dd76850cbc4d1e28826d028b159e9c0a9bd11ad29a3b821f478d1 WatchSource:0}: Error finding container e8f742231d1dd76850cbc4d1e28826d028b159e9c0a9bd11ad29a3b821f478d1: Status 404 returned error can't find the container with id e8f742231d1dd76850cbc4d1e28826d028b159e9c0a9bd11ad29a3b821f478d1
Nov 03 19:47:18.581546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.581508    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:18.581648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.581586    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:18.581648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:18.581621    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:19.126461 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.122580    4531 generic.go:356] "Generic (PLEG): container finished" podID="30ca9921-ca34-465e-8f96-cd15ad34e163" containerID="c49fbba7f272ced5bceca820aee11cb53eff1272744a68d5c9679c570b8b7f38" exitCode=0
Nov 03 19:47:19.126461 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.122845    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6" event={"ID":"30ca9921-ca34-465e-8f96-cd15ad34e163","Type":"ContainerDied","Data":"c49fbba7f272ced5bceca820aee11cb53eff1272744a68d5c9679c570b8b7f38"}
Nov 03 19:47:19.130220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.130066    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" event={"ID":"7c80951b-0a70-422b-acab-00b767b5304f","Type":"ContainerStarted","Data":"059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d"}
Nov 03 19:47:19.130220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.130146    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" event={"ID":"7c80951b-0a70-422b-acab-00b767b5304f","Type":"ContainerStarted","Data":"e8f742231d1dd76850cbc4d1e28826d028b159e9c0a9bd11ad29a3b821f478d1"}
Nov 03 19:47:19.206436 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.206382    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" podStartSLOduration=2.206366291 podStartE2EDuration="2.206366291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:19.191328253 +0000 UTC m=+952.461451550" watchObservedRunningTime="2026-11-03 19:47:19.206366291 +0000 UTC m=+952.476489586"
Nov 03 19:47:19.299100 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.299062    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:19.309560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.309518    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 19:47:19.309912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.309876    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 19:47:19.433530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.433486    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:19.589490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.579030    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:19.601481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.601456    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:19.730265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.729965    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-openshift-ca\") pod \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") "
Nov 03 19:47:19.730265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.730119    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-xks-ca\") pod \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") "
Nov 03 19:47:19.730265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.730214    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-kube-api-access-bj8b9\" (UniqueName: \"kubernetes.io/projected/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-kube-api-access-bj8b9\") pod \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\" (UID: \"09a33b82-5427-41cb-9cd9-f2413e4e4dfb\") "
Nov 03 19:47:19.795664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.792437    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-openshift-ca" pod "09a33b82-5427-41cb-9cd9-f2413e4e4dfb" (UID: "09a33b82-5427-41cb-9cd9-f2413e4e4dfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:19.827203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.815769    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:19.874202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.873442    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:19.998304 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.978379    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-kube-api-access-bj8b9" pod "09a33b82-5427-41cb-9cd9-f2413e4e4dfb" (UID: "09a33b82-5427-41cb-9cd9-f2413e4e4dfb"). InnerVolumeSpecName "kube-api-access-bj8b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:19.998304 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:19.982307    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-xks-ca" pod "09a33b82-5427-41cb-9cd9-f2413e4e4dfb" (UID: "09a33b82-5427-41cb-9cd9-f2413e4e4dfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:20.022204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.021068    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:20.022204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.021107    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj8b9\" (UniqueName: \"kubernetes.io/projected/09a33b82-5427-41cb-9cd9-f2413e4e4dfb-kube-api-access-bj8b9\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:20.292697 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.292596    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 19:47:20.294720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.294309    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 19:47:20.297191 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.297147    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"
Nov 03 19:47:20.298344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.298319    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d" event={"ID":"09a33b82-5427-41cb-9cd9-f2413e4e4dfb","Type":"ContainerDied","Data":"42bda34cd039bdd139f73c3be4deba2c376fb64658e82189ee7db07802b89ffd"}
Nov 03 19:47:20.298574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.298561    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42bda34cd039bdd139f73c3be4deba2c376fb64658e82189ee7db07802b89ffd"
Nov 03 19:47:20.419674 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.419647    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:20.447408 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.447360    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:20.599107 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:47:20.599019    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:47:20.654712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.654675    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:20.710210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.708595    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:20.782241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.768350    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:20.908279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.906796    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ca9921-ca34-465e-8f96-cd15ad34e163-kube-api-access-8gvsg\" (UniqueName: \"kubernetes.io/projected/30ca9921-ca34-465e-8f96-cd15ad34e163-kube-api-access-8gvsg\") pod \"30ca9921-ca34-465e-8f96-cd15ad34e163\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") "
Nov 03 19:47:20.917841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.916418    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-xks-ca\") pod \"30ca9921-ca34-465e-8f96-cd15ad34e163\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") "
Nov 03 19:47:20.917841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.916516    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-openshift-ca\") pod \"30ca9921-ca34-465e-8f96-cd15ad34e163\" (UID: \"30ca9921-ca34-465e-8f96-cd15ad34e163\") "
Nov 03 19:47:20.923468 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.921753    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-openshift-ca" pod "30ca9921-ca34-465e-8f96-cd15ad34e163" (UID: "30ca9921-ca34-465e-8f96-cd15ad34e163"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:20.948619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.943365    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ca9921-ca34-465e-8f96-cd15ad34e163-kube-api-access-8gvsg" pod "30ca9921-ca34-465e-8f96-cd15ad34e163" (UID: "30ca9921-ca34-465e-8f96-cd15ad34e163"). InnerVolumeSpecName "kube-api-access-8gvsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:20.976517 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:20.969522    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-xks-ca" pod "30ca9921-ca34-465e-8f96-cd15ad34e163" (UID: "30ca9921-ca34-465e-8f96-cd15ad34e163"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:21.022720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:21.022207    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gvsg\" (UniqueName: \"kubernetes.io/projected/30ca9921-ca34-465e-8f96-cd15ad34e163-kube-api-access-8gvsg\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:21.022720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:21.022241    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:21.022720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:21.022256    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ca9921-ca34-465e-8f96-cd15ad34e163-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:21.273801 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:21.273717    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 19:47:21.273801 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:21.273780    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 19:47:21.301197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:21.294727    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:21.344734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:21.338598    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"
Nov 03 19:47:21.344734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:21.339790    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6" event={"ID":"30ca9921-ca34-465e-8f96-cd15ad34e163","Type":"ContainerDied","Data":"367111b51e627d06fc577760daa1c0c58dff251fa0b8b63f31314f990f222e94"}
Nov 03 19:47:21.344734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:21.339846    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="367111b51e627d06fc577760daa1c0c58dff251fa0b8b63f31314f990f222e94"
Nov 03 19:47:21.985192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:21.985141    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:22.274026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:22.273936    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 19:47:22.274026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:22.273995    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 19:47:22.902976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:22.902927    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:23.089063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:23.089011    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:23.274262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:23.274066    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 19:47:23.274539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:23.274496    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 19:47:23.291507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:23.291440    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:23.475478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:23.475441    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:24.026081 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:24.026024    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:24.273747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:24.273717    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 19:47:24.273896 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:24.273769    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 19:47:25.100719 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:25.100675    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:25.273868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:25.273829    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 19:47:25.274020 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:25.273886    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 19:47:25.562512 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:25.562477    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:25.875744 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:25.875654    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:26.274046 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:26.273998    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 19:47:26.274524 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:26.274126    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 19:47:26.539957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:26.539855    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:27.164111 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.164070    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:27.279144 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.279105    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:27.279693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.279656    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:27.280650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.280627    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 19:47:27.388240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.388209    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"]
Nov 03 19:47:27.388938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.388924    4531 cpu_manager.go:414] "containerMap: removing container" podUID="09a33b82-5427-41cb-9cd9-f2413e4e4dfb" containerName="revoke-keys"
Nov 03 19:47:27.389017 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.388939    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a33b82-5427-41cb-9cd9-f2413e4e4dfb" containerName="revoke-keys"
Nov 03 19:47:27.389017 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.388988    4531 cpu_manager.go:414] "containerMap: removing container" podUID="30ca9921-ca34-465e-8f96-cd15ad34e163" containerName="revoke-keys"
Nov 03 19:47:27.389017 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.388995    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ca9921-ca34-465e-8f96-cd15ad34e163" containerName="revoke-keys"
Nov 03 19:47:27.389114 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.389088    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ca9921-ca34-465e-8f96-cd15ad34e163" containerName="revoke-keys"
Nov 03 19:47:27.389114 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.389098    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a33b82-5427-41cb-9cd9-f2413e4e4dfb" containerName="revoke-keys"
Nov 03 19:47:27.394416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.394390    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:27.397909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.397885    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:47:27.398105 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.397909    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:47:27.402245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.402221    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"]
Nov 03 19:47:27.559850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.559769    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks9wq\" (UniqueName: \"kubernetes.io/projected/e0507087-9b46-4812-9dbb-14e5df1e2046-kube-api-access-ks9wq\") pod \"maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:27.560042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.560028    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:27.560259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.560228    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:27.614135 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.614101    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"]
Nov 03 19:47:27.619601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.619573    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:27.628353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.628323    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"]
Nov 03 19:47:27.662096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.662058    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks9wq\" (UniqueName: \"kubernetes.io/projected/e0507087-9b46-4812-9dbb-14e5df1e2046-kube-api-access-ks9wq\") pod \"maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:27.662308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.662287    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:27.662447 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.662428    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:27.663146 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.663119    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:27.664773 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.664752    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:27.669109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.669035    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks9wq\" (UniqueName: \"kubernetes.io/projected/e0507087-9b46-4812-9dbb-14e5df1e2046-kube-api-access-ks9wq\") pod \"maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:27.719914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.719885    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:27.764412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.764376    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:27.764721 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.764694    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:27.764846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.764816    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwh6\" (UniqueName: \"kubernetes.io/projected/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-kube-api-access-dfwh6\") pod \"maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:27.868097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.868056    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:27.868417 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.868393    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:27.868568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.868532    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwh6\" (UniqueName: \"kubernetes.io/projected/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-kube-api-access-dfwh6\") pod \"maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:27.869647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.869620    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:27.875249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.875224    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:27.877703 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.877663    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwh6\" (UniqueName: \"kubernetes.io/projected/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-kube-api-access-dfwh6\") pod \"maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:27.943816 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:27.943787    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:28.041676 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:28.041639    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:28.042384 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:28.042357    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0507087_9b46_4812_9dbb_14e5df1e2046.slice/crio-0a39b3407adf1dfcaef71434a67ba88cf53db6b8cae1cde6680b94554dc0ab8a WatchSource:0}: Error finding container 0a39b3407adf1dfcaef71434a67ba88cf53db6b8cae1cde6680b94554dc0ab8a: Status 404 returned error can't find the container with id 0a39b3407adf1dfcaef71434a67ba88cf53db6b8cae1cde6680b94554dc0ab8a
Nov 03 19:47:28.062212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:28.044840    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"]
Nov 03 19:47:28.231593 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:28.231562    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a6dc87_9edc_4424_b2b0_fe804d8b9bf3.slice/crio-ca62c586360b4108761dda3a08d9822678fc44f77386e6f20ae87c8c8a76228b WatchSource:0}: Error finding container ca62c586360b4108761dda3a08d9822678fc44f77386e6f20ae87c8c8a76228b: Status 404 returned error can't find the container with id ca62c586360b4108761dda3a08d9822678fc44f77386e6f20ae87c8c8a76228b
Nov 03 19:47:28.233940 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:28.233915    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"]
Nov 03 19:47:28.468593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:28.468550    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2" event={"ID":"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3","Type":"ContainerStarted","Data":"ca62c586360b4108761dda3a08d9822678fc44f77386e6f20ae87c8c8a76228b"}
Nov 03 19:47:28.473129 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:28.473100    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl" event={"ID":"e0507087-9b46-4812-9dbb-14e5df1e2046","Type":"ContainerStarted","Data":"0a39b3407adf1dfcaef71434a67ba88cf53db6b8cae1cde6680b94554dc0ab8a"}
Nov 03 19:47:29.481810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:29.481774    4531 generic.go:356] "Generic (PLEG): container finished" podID="e0507087-9b46-4812-9dbb-14e5df1e2046" containerID="f53f7a497a8f91b32b9f630ec40eb6995997df6641adbe53660ad828e6f296b1" exitCode=0
Nov 03 19:47:29.482263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:29.481856    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl" event={"ID":"e0507087-9b46-4812-9dbb-14e5df1e2046","Type":"ContainerDied","Data":"f53f7a497a8f91b32b9f630ec40eb6995997df6641adbe53660ad828e6f296b1"}
Nov 03 19:47:29.484070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:29.484040    4531 generic.go:356] "Generic (PLEG): container finished" podID="e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3" containerID="7bdc6c802ee84a02a0b5039e98498f9aeb82d6e99aafdbdfa1e7b7cc9dab2508" exitCode=0
Nov 03 19:47:29.484225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:29.484088    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2" event={"ID":"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3","Type":"ContainerDied","Data":"7bdc6c802ee84a02a0b5039e98498f9aeb82d6e99aafdbdfa1e7b7cc9dab2508"}
Nov 03 19:47:29.562418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:29.562383    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:30.250685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:30.250645    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:30.432237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:30.430228    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:30.818263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:30.814107    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:30.897198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:30.862266    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.029198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.028361    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.064257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.063282    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"]
Nov 03 19:47:31.118715 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.087305    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.126390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.126327    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.138783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.138743    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.162199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.159016    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:47:31.218195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.214588    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"]
Nov 03 19:47:31.218195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.214723    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:31.438445 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.438376    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.538593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.538541    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:31.542201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.539225    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7987\" (UniqueName: \"kubernetes.io/projected/0f7cd946-a247-447b-8812-4b31be2f12d5-kube-api-access-f7987\") pod \"maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:31.557677 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.557633    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:31.681294 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.657900    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.713281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.701730    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7987\" (UniqueName: \"kubernetes.io/projected/0f7cd946-a247-447b-8812-4b31be2f12d5-kube-api-access-f7987\") pod \"maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:31.713281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.701778    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:31.713281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.701930    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:31.713281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.705641    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:31.786285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.786213    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:31.794202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.793070    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7987\" (UniqueName: \"kubernetes.io/projected/0f7cd946-a247-447b-8812-4b31be2f12d5-kube-api-access-f7987\") pod \"maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:31.820325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.820281    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:47:31.825196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.822007    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:47:31.954224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.932441    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:47:31.976255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:31.955897    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:32.108196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.097117    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:32.108196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.099204    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:32.108196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.106857    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:47:32.166198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.162010    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:32.188198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.187932    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:47:32.346293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.217618    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:32.346293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.218422    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:47:32.346293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.218948    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:47:32.346293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.219241    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:47:32.346293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.219337    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:47:32.346293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.219393    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:47:32.456202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.400837    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:47:32.456202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.419464    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:47:32.456202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.449767    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:32.532275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.506561    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:47:32.683205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.677637    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:32.752231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.746750    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:47:32.823479 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.823430    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:47:32.910448 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.910398    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:32.950396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.950291    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:33.017200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.952866    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:47:33.037209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.992393    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6" podUID="c731a1ef-3377-4475-82cb-213ba543c002" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:33.064862 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:32.993706    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt" podUID="6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded"
Nov 03 19:47:33.099122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.099078    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:33.099472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.099446    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:33.242162 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.241942    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:33.243024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.242896    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:47:33.243619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.243592    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:47:33.302658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.276422    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:33.416195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.416118    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:47:33.597258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.587145    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 03 19:47:33.938293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.938239    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:47:33.942200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.941118    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:47:33.989229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.989189    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.010544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:33.992234    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.109220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.103330    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:34.121396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.112870    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6" podUID="c731a1ef-3377-4475-82cb-213ba543c002" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.121396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.113193    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt" podUID="6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.121396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.113432    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.121396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.114635    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:34.148224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.146103    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:34.148224 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.146189    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:34.293317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.293269    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:34.312759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.053271    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.316788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.316747    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.368418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.368371    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:34.395142 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.395084    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.530257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.508438    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:34.640856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.551960    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:34.641090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.641047    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:34.641224 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:47:34.617817    4531 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.6s"
Nov 03 19:47:34.661954 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.661913    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:47:34.695076 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.693972    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:47:34.771137 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.771099    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:47:34.812262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.783331    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:47:34.992565 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:34.992504    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.053254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.051622    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:47:35.073263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.053848    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:47:35.073263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.073102    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.076694 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.074487    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.097852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.097710    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:35.097852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.097771    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.177993 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.177953    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:35.178619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.178125    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.270710 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.270664    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.271043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.271004    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.298249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.292470    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.344253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.320295    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.363205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.329437    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.363205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.294273    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.394231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.386844    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:47:35.417216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.413799    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.585330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.585296    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:35.591600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.591566    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:47:35.591865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.591848    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hwtt"
Nov 03 19:47:35.591994 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.591982    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6"
Nov 03 19:47:35.690204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.688282    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.786200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.777685    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2" event={"ID":"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3","Type":"ContainerDied","Data":"ca62c586360b4108761dda3a08d9822678fc44f77386e6f20ae87c8c8a76228b"}
Nov 03 19:47:35.786200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.777750    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca62c586360b4108761dda3a08d9822678fc44f77386e6f20ae87c8c8a76228b"
Nov 03 19:47:35.802200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.801106    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.865199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.849645    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-openshift-ca\") pod \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") "
Nov 03 19:47:35.865199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.849706    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-kube-api-access-dfwh6\" (UniqueName: \"kubernetes.io/projected/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-kube-api-access-dfwh6\") pod \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") "
Nov 03 19:47:35.865199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.849970    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-xks-ca\") pod \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\" (UID: \"e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3\") "
Nov 03 19:47:35.921507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.889435    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-openshift-ca" pod "e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3" (UID: "e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:35.923119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.923082    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:35.957022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.956977    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-openshift-ca\") pod \"e0507087-9b46-4812-9dbb-14e5df1e2046\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") "
Nov 03 19:47:35.957324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.957300    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0507087-9b46-4812-9dbb-14e5df1e2046-kube-api-access-ks9wq\" (UniqueName: \"kubernetes.io/projected/e0507087-9b46-4812-9dbb-14e5df1e2046-kube-api-access-ks9wq\") pod \"e0507087-9b46-4812-9dbb-14e5df1e2046\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") "
Nov 03 19:47:35.957498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.957481    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-xks-ca\") pod \"e0507087-9b46-4812-9dbb-14e5df1e2046\" (UID: \"e0507087-9b46-4812-9dbb-14e5df1e2046\") "
Nov 03 19:47:35.964366 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:35.958908    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:36.037206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.023725    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-openshift-ca" pod "e0507087-9b46-4812-9dbb-14e5df1e2046" (UID: "e0507087-9b46-4812-9dbb-14e5df1e2046"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:36.079120 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.079076    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.083196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.079936    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.159199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.125717    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.182677 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.182574    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:47:36.183740 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.183708    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:36.196156 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.195582    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.197596 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.197566    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:36.207189 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.207141    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-xks-ca" pod "e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3" (UID: "e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:36.257580 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.257538    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0507087-9b46-4812-9dbb-14e5df1e2046-kube-api-access-ks9wq" pod "e0507087-9b46-4812-9dbb-14e5df1e2046" (UID: "e0507087-9b46-4812-9dbb-14e5df1e2046"). InnerVolumeSpecName "kube-api-access-ks9wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:36.292097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.292060    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-xks-ca" pod "e0507087-9b46-4812-9dbb-14e5df1e2046" (UID: "e0507087-9b46-4812-9dbb-14e5df1e2046"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:36.405333 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.405278    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:47:36.423230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.405928    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.434247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.415280    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Nov 03 19:47:36.452202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.415383    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.452202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.415409    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.452202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.436889    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.452202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.438764    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks9wq\" (UniqueName: \"kubernetes.io/projected/e0507087-9b46-4812-9dbb-14e5df1e2046-kube-api-access-ks9wq\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:36.452202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.438794    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0507087-9b46-4812-9dbb-14e5df1e2046-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:36.452202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.438809    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:36.553285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.551769    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-kube-api-access-dfwh6" pod "e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3" (UID: "e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3"). InnerVolumeSpecName "kube-api-access-dfwh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:36.640606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.600943    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfwh6\" (UniqueName: \"kubernetes.io/projected/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3-kube-api-access-dfwh6\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:36.725256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:36.695542    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:37.183969 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.183917    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:47:37.562385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.553588    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:47:37.637341 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.637291    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:47:37.637973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.637939    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:47:37.638998 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.638969    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:37.741258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.720571    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:37.830896 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.830570    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:47:37.898020 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.866294    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:47:38.058830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.892307    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:47:38.079355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.900267    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:47:38.080471 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.901259    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:47:38.081212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:37.901292    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:47:38.253201 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:47:38.249344    4531 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.143s"
Nov 03 19:47:38.341957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.341859    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"
Nov 03 19:47:38.512197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.346930    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"
Nov 03 19:47:38.546197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.537968    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:47:38.591264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.346953    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl" event={"ID":"e0507087-9b46-4812-9dbb-14e5df1e2046","Type":"ContainerDied","Data":"0a39b3407adf1dfcaef71434a67ba88cf53db6b8cae1cde6680b94554dc0ab8a"}
Nov 03 19:47:38.591561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.591542    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a39b3407adf1dfcaef71434a67ba88cf53db6b8cae1cde6680b94554dc0ab8a"
Nov 03 19:47:38.701650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.701593    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:47:38.702456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.702422    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:47:38.706195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.702984    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:47:38.795213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.772670    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:38.832593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.818639    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": context deadline exceeded"
Nov 03 19:47:38.832593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.819907    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:38.881204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.865453    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:38.914484 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.914443    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:38.926413 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.926370    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7n48w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:38.926695 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.926662    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" podUID="bd352072-e76d-4d10-93a1-a71b2e0495ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:38.961313 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.932470    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Nov 03 19:47:38.961668 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:38.961616    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:39.085384 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:39.085321    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:39.085818 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:39.085776    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:47:39.086160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:39.086136    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:39.109089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:39.109031    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:47:39.583754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:39.572080    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:39.920221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:39.908308    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:39.992063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:39.969872    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mw5r6" podUID="c731a1ef-3377-4475-82cb-213ba543c002" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded"
Nov 03 19:47:40.036452 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.036394    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:40.082200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.075405    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:47:40.219476 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:47:40.219360    4531 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0507087_9b46_4812_9dbb_14e5df1e2046.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 19:47:40.684206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.227503    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:47:40.847640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.838276    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:40.939247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.244413    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:47:40.939247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.264328    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:40.939247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.202705    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:47:40.939247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.932089    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:47:40.939247 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:47:40.320062    4531 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7cd946_a247_447b_8812_4b31be2f12d5.slice/crio-conmon-9fd50fe798186a802b97ec9c73d1445536fb1f5e97f871456fe49f6ede2aed6a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:47:40.939957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.306723    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:47:40.941428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.383721    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:47:40.968199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.383680    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:47:40.968199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.427651    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:40.969954 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.017456    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Nov 03 19:47:41.083400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:40.244359    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:47:41.083400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.041374    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:41.083400 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:47:41.080907    4531 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.214s"
Nov 03 19:47:41.094251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.087801    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:41.316065 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.315972    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:41.610322 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.610225    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv" event={"ID":"0f7cd946-a247-447b-8812-4b31be2f12d5","Type":"ContainerStarted","Data":"d4de220b1fc62cbeba9a943bff9cfae3211749853635f304ea552427cb920a01"}
Nov 03 19:47:41.716555 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.716491    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:41.724784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.724322    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:41.800018 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.799976    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:41.947210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.938389    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:41.947210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.938788    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:47:41.947210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.939101    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:47:41.947210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.940510    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:41.947210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.941662    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:41.947210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.942993    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:41.970350 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.961351    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:41.970350 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:41.961754    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:47:42.046861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:42.038290    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:42.172651 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:42.166726    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:47:42.199266 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:42.170345    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:42.263367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:42.243764    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:47:42.652261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:42.626846    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:47:42.728637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:42.728032    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:47:42.944251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:42.825307    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:42.944251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:42.940679    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:42.944251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:42.943408    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:47:43.230553 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.230456    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:47:43.280248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.279956    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:43.447199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.439293    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:43.483610 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.448116    4531 generic.go:356] "Generic (PLEG): container finished" podID="0f7cd946-a247-447b-8812-4b31be2f12d5" containerID="9fd50fe798186a802b97ec9c73d1445536fb1f5e97f871456fe49f6ede2aed6a" exitCode=0
Nov 03 19:47:43.483610 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.448713    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:43.483610 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.450221    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:43.595869 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.595828    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv" event={"ID":"0f7cd946-a247-447b-8812-4b31be2f12d5","Type":"ContainerDied","Data":"9fd50fe798186a802b97ec9c73d1445536fb1f5e97f871456fe49f6ede2aed6a"}
Nov 03 19:47:43.630600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.628499    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:47:43.630600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.628572    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:47:43.630600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.628624    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:47:43.630600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.628673    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:47:43.630600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.628327    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:43.829195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.829093    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:47:43.845749 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.845699    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:47:43.856056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.856021    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:47:43.871826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.849450    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:43.873887 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.873862    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"]
Nov 03 19:47:43.897000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.896974    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:47:43.945921 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.945892    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:47:43.998206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:43.998160    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"]
Nov 03 19:47:44.020833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.020804    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"]
Nov 03 19:47:44.030271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.030231    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"]
Nov 03 19:47:44.075377 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.073793    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.090251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.082055    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"]
Nov 03 19:47:44.090251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.082285    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf" podUID="365b7002-7797-48e2-b7bd-d8ecb75fb969" containerName="maas-api" containerID="cri-o://82426fd76d62e1f8bc3a3bffb9dc23c894a87ad942855bcfed0502dd7a875ecf" gracePeriod=30
Nov 03 19:47:44.105764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.105516    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"]
Nov 03 19:47:44.115618 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.115580    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbfcf47a-xmcjl"]
Nov 03 19:47:44.322440 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.322405    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.492495 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.492454    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.492989 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.492962    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"]
Nov 03 19:47:44.495035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.495005    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3" containerName="revoke-keys"
Nov 03 19:47:44.495035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.495034    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3" containerName="revoke-keys"
Nov 03 19:47:44.495307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.495054    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e0507087-9b46-4812-9dbb-14e5df1e2046" containerName="revoke-keys"
Nov 03 19:47:44.495307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.495063    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0507087-9b46-4812-9dbb-14e5df1e2046" containerName="revoke-keys"
Nov 03 19:47:44.495414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.495307    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3" containerName="revoke-keys"
Nov 03 19:47:44.495414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.495320    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0507087-9b46-4812-9dbb-14e5df1e2046" containerName="revoke-keys"
Nov 03 19:47:44.501720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.501222    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.503359 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.503333    4531 generic.go:356] "Generic (PLEG): container finished" podID="365b7002-7797-48e2-b7bd-d8ecb75fb969" containerID="82426fd76d62e1f8bc3a3bffb9dc23c894a87ad942855bcfed0502dd7a875ecf" exitCode=0
Nov 03 19:47:44.503982 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.503953    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf" event={"ID":"365b7002-7797-48e2-b7bd-d8ecb75fb969","Type":"ContainerDied","Data":"82426fd76d62e1f8bc3a3bffb9dc23c894a87ad942855bcfed0502dd7a875ecf"}
Nov 03 19:47:44.514560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.508114    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"]
Nov 03 19:47:44.514560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.511939    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c96ee058-gw-openshift-default-dockercfg-4c5sh\""
Nov 03 19:47:44.605703 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.605392    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:44.748511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.708561    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-maas-api-tls\") pod \"365b7002-7797-48e2-b7bd-d8ecb75fb969\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") "
Nov 03 19:47:44.748511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.708716    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-metrics-certs\" (UniqueName: \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-metrics-certs\") pod \"365b7002-7797-48e2-b7bd-d8ecb75fb969\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") "
Nov 03 19:47:44.748511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.708811    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365b7002-7797-48e2-b7bd-d8ecb75fb969-kube-api-access-d8wp9\" (UniqueName: \"kubernetes.io/projected/365b7002-7797-48e2-b7bd-d8ecb75fb969-kube-api-access-d8wp9\") pod \"365b7002-7797-48e2-b7bd-d8ecb75fb969\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") "
Nov 03 19:47:44.748511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.708937    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365b7002-7797-48e2-b7bd-d8ecb75fb969-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/365b7002-7797-48e2-b7bd-d8ecb75fb969-openshift-service-ca\") pod \"365b7002-7797-48e2-b7bd-d8ecb75fb969\" (UID: \"365b7002-7797-48e2-b7bd-d8ecb75fb969\") "
Nov 03 19:47:44.748511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.709147    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a7070e-597a-4d4f-9f62-ea93932fffd3-istiod-ca-cert\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.748511 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.730716    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.807195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.796887    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-token\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.807195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.796938    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-data\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.807195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.797047    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-podinfo\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.807195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.797127    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-credential-socket\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.807195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.805265    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxt7b\" (UniqueName: \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-kube-api-access-lxt7b\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.807195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.805462    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-socket\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.807195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.805516    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-certs\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.807195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.805575    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-envoy\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.830670    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365b7002-7797-48e2-b7bd-d8ecb75fb969-openshift-service-ca" pod "365b7002-7797-48e2-b7bd-d8ecb75fb969" (UID: "365b7002-7797-48e2-b7bd-d8ecb75fb969"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.900008    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.909967    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-socket\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.910030    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-certs\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.910585    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-socket\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.910644    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-envoy\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.910762    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a7070e-597a-4d4f-9f62-ea93932fffd3-istiod-ca-cert\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.910892    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-token\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.910920    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-data\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.911018    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-podinfo\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.911090    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-credential-socket\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.911149    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxt7b\" (UniqueName: \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-kube-api-access-lxt7b\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.923538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.911375    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/365b7002-7797-48e2-b7bd-d8ecb75fb969-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:44.950199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.927126    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-data\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.950199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.950136    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-certs\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.972152 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.951014    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a7070e-597a-4d4f-9f62-ea93932fffd3-istiod-ca-cert\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:44.980661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:44.973794    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-credential-socket\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:45.072214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.050976    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-token\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:45.111204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.080922    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:45.190905 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.190838    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-metrics-certs" pod "365b7002-7797-48e2-b7bd-d8ecb75fb969" (UID: "365b7002-7797-48e2-b7bd-d8ecb75fb969"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:45.191272 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.191243    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-maas-api-tls" pod "365b7002-7797-48e2-b7bd-d8ecb75fb969" (UID: "365b7002-7797-48e2-b7bd-d8ecb75fb969"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:45.193389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.193364    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365b7002-7797-48e2-b7bd-d8ecb75fb969-kube-api-access-d8wp9" pod "365b7002-7797-48e2-b7bd-d8ecb75fb969" (UID: "365b7002-7797-48e2-b7bd-d8ecb75fb969"). InnerVolumeSpecName "kube-api-access-d8wp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:45.205054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.195338    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxt7b\" (UniqueName: \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-kube-api-access-lxt7b\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:45.205054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.203691    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-podinfo\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:45.205054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.204558    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-envoy\") pod \"e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") " pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:45.219475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.214986    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:45.219475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.218066    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0507087-9b46-4812-9dbb-14e5df1e2046" path="/var/lib/kubelet/pods/e0507087-9b46-4812-9dbb-14e5df1e2046/volumes"
Nov 03 19:47:45.223279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.223108    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"]
Nov 03 19:47:45.223279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.223149    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e17271b-bv5c2"]
Nov 03 19:47:45.291317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.282701    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8wp9\" (UniqueName: \"kubernetes.io/projected/365b7002-7797-48e2-b7bd-d8ecb75fb969-kube-api-access-d8wp9\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:45.291317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.282744    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:45.291317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.282760    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/365b7002-7797-48e2-b7bd-d8ecb75fb969-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:45.399207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.398443    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:45.434279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.434088    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:45.489843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.489806    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7cd946-a247-447b-8812-4b31be2f12d5-kube-api-access-f7987\" (UniqueName: \"kubernetes.io/projected/0f7cd946-a247-447b-8812-4b31be2f12d5-kube-api-access-f7987\") pod \"0f7cd946-a247-447b-8812-4b31be2f12d5\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") "
Nov 03 19:47:45.490007 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.489886    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-xks-ca\") pod \"0f7cd946-a247-447b-8812-4b31be2f12d5\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") "
Nov 03 19:47:45.490147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.490127    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-openshift-ca\") pod \"0f7cd946-a247-447b-8812-4b31be2f12d5\" (UID: \"0f7cd946-a247-447b-8812-4b31be2f12d5\") "
Nov 03 19:47:45.490744 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.490716    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-openshift-ca" pod "0f7cd946-a247-447b-8812-4b31be2f12d5" (UID: "0f7cd946-a247-447b-8812-4b31be2f12d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:45.492562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.492532    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:45.498609 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.496356    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7cd946-a247-447b-8812-4b31be2f12d5-kube-api-access-f7987" pod "0f7cd946-a247-447b-8812-4b31be2f12d5" (UID: "0f7cd946-a247-447b-8812-4b31be2f12d5"). InnerVolumeSpecName "kube-api-access-f7987". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:45.498754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.498602    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-xks-ca" pod "0f7cd946-a247-447b-8812-4b31be2f12d5" (UID: "0f7cd946-a247-447b-8812-4b31be2f12d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:45.529683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.529643    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf" event={"ID":"365b7002-7797-48e2-b7bd-d8ecb75fb969","Type":"ContainerDied","Data":"c8bb34899451a19495e9b72b9087e0bd57b798649e9325009fc2fbcea674771a"}
Nov 03 19:47:45.529846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.529720    4531 scope.go:122] "RemoveContainer" containerID="82426fd76d62e1f8bc3a3bffb9dc23c894a87ad942855bcfed0502dd7a875ecf"
Nov 03 19:47:45.529885 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.529877    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"
Nov 03 19:47:45.535237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.535125    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv" event={"ID":"0f7cd946-a247-447b-8812-4b31be2f12d5","Type":"ContainerDied","Data":"d4de220b1fc62cbeba9a943bff9cfae3211749853635f304ea552427cb920a01"}
Nov 03 19:47:45.535237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.535167    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4de220b1fc62cbeba9a943bff9cfae3211749853635f304ea552427cb920a01"
Nov 03 19:47:45.535412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.535386    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"
Nov 03 19:47:45.588486 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.588454    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"]
Nov 03 19:47:45.597013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.595845    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7987\" (UniqueName: \"kubernetes.io/projected/0f7cd946-a247-447b-8812-4b31be2f12d5-kube-api-access-f7987\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:45.597013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.595869    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7cd946-a247-447b-8812-4b31be2f12d5-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:45.597013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.596494    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cbfcf47a-799d4479c-h5xvf"]
Nov 03 19:47:45.824265 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:45.824228    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a7070e_597a_4d4f_9f62_ea93932fffd3.slice/crio-ba5e080cf211040645bb554e83372f4eab88cc000dd8422a1a96324a27de8c35 WatchSource:0}: Error finding container ba5e080cf211040645bb554e83372f4eab88cc000dd8422a1a96324a27de8c35: Status 404 returned error can't find the container with id ba5e080cf211040645bb554e83372f4eab88cc000dd8422a1a96324a27de8c35
Nov 03 19:47:45.826287 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.826265    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"]
Nov 03 19:47:45.827136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.827101    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:45.827226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.827209    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:45.827296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:45.827250    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:46.061488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:46.061440    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:46.513279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:46.512186    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"]
Nov 03 19:47:46.513279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:46.512495    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr" podUID="7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" containerName="istio-proxy" containerID="cri-o://18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719" gracePeriod=30
Nov 03 19:47:46.548325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:46.548290    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7" event={"ID":"24a7070e-597a-4d4f-9f62-ea93932fffd3","Type":"ContainerStarted","Data":"9facc79e9d4af4a75b881d5f163186f5f41507504ca4b69009e4b8d6d78a49f3"}
Nov 03 19:47:46.548525 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:46.548347    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7" event={"ID":"24a7070e-597a-4d4f-9f62-ea93932fffd3","Type":"ContainerStarted","Data":"ba5e080cf211040645bb554e83372f4eab88cc000dd8422a1a96324a27de8c35"}
Nov 03 19:47:46.568295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:46.568242    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7" podStartSLOduration=2.568229181 podStartE2EDuration="2.568229181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:46.564233311 +0000 UTC m=+979.834356604" watchObservedRunningTime="2026-11-03 19:47:46.568229181 +0000 UTC m=+979.838352475"
Nov 03 19:47:46.585312 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:46.585271    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:46.943706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:46.943664    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365b7002-7797-48e2-b7bd-d8ecb75fb969" path="/var/lib/kubelet/pods/365b7002-7797-48e2-b7bd-d8ecb75fb969/volumes"
Nov 03 19:47:46.946152 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:46.946121    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3" path="/var/lib/kubelet/pods/e0a6dc87-9edc-4424-b2b0-fe804d8b9bf3/volumes"
Nov 03 19:47:46.959825 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:46.959730    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.204121 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.203967    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.434578 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.434552    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:47.442123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.441750    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:47.511573 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.511493    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"]
Nov 03 19:47:47.512663 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.512640    4531 cpu_manager.go:414] "containerMap: removing container" podUID="365b7002-7797-48e2-b7bd-d8ecb75fb969" containerName="maas-api"
Nov 03 19:47:47.512663 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.512661    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365b7002-7797-48e2-b7bd-d8ecb75fb969" containerName="maas-api"
Nov 03 19:47:47.512850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.512701    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0f7cd946-a247-447b-8812-4b31be2f12d5" containerName="revoke-keys"
Nov 03 19:47:47.512850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.512709    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7cd946-a247-447b-8812-4b31be2f12d5" containerName="revoke-keys"
Nov 03 19:47:47.512956 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.512895    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="365b7002-7797-48e2-b7bd-d8ecb75fb969" containerName="maas-api"
Nov 03 19:47:47.512956 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.512913    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f7cd946-a247-447b-8812-4b31be2f12d5" containerName="revoke-keys"
Nov 03 19:47:47.531770 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.531713    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"]
Nov 03 19:47:47.531960 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.531926    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.537689 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.535800    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b3a1058c-openshift-default-dockercfg-28h5x\""
Nov 03 19:47:47.574777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.574746    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:47.575867 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.575845    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:47:47.611043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.611008    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.642236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.639900    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-certs\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.642236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.640124    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-credential-socket\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.642236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.640275    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-envoy\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.642236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.640370    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-token\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.642236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.640412    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94wc6\" (UniqueName: \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-kube-api-access-94wc6\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.642236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.640565    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-podinfo\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.642236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.640675    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-socket\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.642236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.640752    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istiod-ca-cert\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.642236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.640839    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-data\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743017    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-data\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743220    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-certs\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743405 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743389    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-credential-socket\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743440 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743430    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-envoy\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743473    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-data\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743479    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-token\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743529    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-certs\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743639 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743558    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94wc6\" (UniqueName: \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-kube-api-access-94wc6\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743662    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-podinfo\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743752 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743699    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-credential-socket\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743806 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743771    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-socket\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.743859 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.743802    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istiod-ca-cert\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.744086 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.744064    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-socket\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.744391 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.744371    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istiod-ca-cert\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.745941 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.745914    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-envoy\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.746042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.745920    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-podinfo\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.753820 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.753790    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-token\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.753927 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.753887    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94wc6\" (UniqueName: \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-kube-api-access-94wc6\") pod \"e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") " pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:47.840275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.835229    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.899198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:47.898611    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:48.493163 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:48.493131    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d3e888_4314_4c8a_b4a4_cad39c13ee1a.slice/crio-1b06d26d1d7b50e0b726f28808b125523e9800117d5e488d7f8742b02e06f5b5 WatchSource:0}: Error finding container 1b06d26d1d7b50e0b726f28808b125523e9800117d5e488d7f8742b02e06f5b5: Status 404 returned error can't find the container with id 1b06d26d1d7b50e0b726f28808b125523e9800117d5e488d7f8742b02e06f5b5
Nov 03 19:47:48.495684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.495658    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"]
Nov 03 19:47:48.496423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.496393    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:48.496521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.496461    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:48.496521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.496488    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:48.501996 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.501969    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:48.639755 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.637491    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5" event={"ID":"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a","Type":"ContainerStarted","Data":"1b06d26d1d7b50e0b726f28808b125523e9800117d5e488d7f8742b02e06f5b5"}
Nov 03 19:47:48.652623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.652518    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:48.940134 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.940098    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"]
Nov 03 19:47:48.960808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.960636    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"]
Nov 03 19:47:48.960808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.960759    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:47:48.970899 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.970268    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c96ee058\""
Nov 03 19:47:48.987262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.985406    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"]
Nov 03 19:47:49.001968 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:48.999277    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml" podUID="38448d63-7dad-48b1-8373-b72e8ede03af" containerName="payload-processing" containerID="cri-o://d931dcf3840fe03d5a9acdd6f7dd5354116f371dbe4619ccad22ecc6ee940cc5" gracePeriod=25
Nov 03 19:47:49.004327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.003926    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:49.007409 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.006962    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"]
Nov 03 19:47:49.007409 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.007074    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:47:49.013903 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.013166    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c96ee058-dockercfg-8bgtr\""
Nov 03 19:47:49.023366 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.021765    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz" podUID="93adf525-5fd1-4beb-bc15-990bbf213b5c" containerName="payload-pre-processing" containerID="cri-o://f37f791d03b5658e560e068e4dd3f8b70d7ae141f7540a0aa9f0993055be5869" gracePeriod=25
Nov 03 19:47:49.031406 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.030806    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv" podUID="91501c4c-a6cc-479b-b660-17a0604c7346" containerName="payload-processing" containerID="cri-o://bdb46d723768ca21340767fcbf00661e9122e8f9cb2d65726b0b042853dac682" gracePeriod=25
Nov 03 19:47:49.069821 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.069747    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba12cce-ee41-46d3-a181-b01838b310cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76\" (UID: \"bba12cce-ee41-46d3-a181-b01838b310cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:47:49.070195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.069800    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58lsv\" (UniqueName: \"kubernetes.io/projected/f7d1aab5-730f-4393-a956-d721f1a7fcc0-kube-api-access-58lsv\") pod \"payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm\" (UID: \"f7d1aab5-730f-4393-a956-d721f1a7fcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:47:49.070669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.070336    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghwf6\" (UniqueName: \"kubernetes.io/projected/bba12cce-ee41-46d3-a181-b01838b310cd-kube-api-access-ghwf6\") pod \"payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76\" (UID: \"bba12cce-ee41-46d3-a181-b01838b310cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:47:49.070669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.070520    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d1aab5-730f-4393-a956-d721f1a7fcc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm\" (UID: \"f7d1aab5-730f-4393-a956-d721f1a7fcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:47:49.157100 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.148494    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:49.238842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.228077    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d1aab5-730f-4393-a956-d721f1a7fcc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm\" (UID: \"f7d1aab5-730f-4393-a956-d721f1a7fcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:47:49.238842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.228413    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba12cce-ee41-46d3-a181-b01838b310cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76\" (UID: \"bba12cce-ee41-46d3-a181-b01838b310cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:47:49.238842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.228451    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58lsv\" (UniqueName: \"kubernetes.io/projected/f7d1aab5-730f-4393-a956-d721f1a7fcc0-kube-api-access-58lsv\") pod \"payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm\" (UID: \"f7d1aab5-730f-4393-a956-d721f1a7fcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:47:49.238842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.228499    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghwf6\" (UniqueName: \"kubernetes.io/projected/bba12cce-ee41-46d3-a181-b01838b310cd-kube-api-access-ghwf6\") pod \"payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76\" (UID: \"bba12cce-ee41-46d3-a181-b01838b310cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:47:49.238842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.229829    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d1aab5-730f-4393-a956-d721f1a7fcc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm\" (UID: \"f7d1aab5-730f-4393-a956-d721f1a7fcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:47:49.238842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.237824    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba12cce-ee41-46d3-a181-b01838b310cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76\" (UID: \"bba12cce-ee41-46d3-a181-b01838b310cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:47:49.474481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.474357    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghwf6\" (UniqueName: \"kubernetes.io/projected/bba12cce-ee41-46d3-a181-b01838b310cd-kube-api-access-ghwf6\") pod \"payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76\" (UID: \"bba12cce-ee41-46d3-a181-b01838b310cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:47:49.498295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.489841    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58lsv\" (UniqueName: \"kubernetes.io/projected/f7d1aab5-730f-4393-a956-d721f1a7fcc0-kube-api-access-58lsv\") pod \"payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm\" (UID: \"f7d1aab5-730f-4393-a956-d721f1a7fcc0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:47:49.645672 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.645329    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:47:49.668532 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.667737    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5" event={"ID":"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a","Type":"ContainerStarted","Data":"193c6959f1c790fe85f81a1ff398cfddb086bc5ee4035a4f11fd4497d441fbac"}
Nov 03 19:47:49.673462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.673432    4531 generic.go:356] "Generic (PLEG): container finished" podID="93adf525-5fd1-4beb-bc15-990bbf213b5c" containerID="f37f791d03b5658e560e068e4dd3f8b70d7ae141f7540a0aa9f0993055be5869" exitCode=0
Nov 03 19:47:49.677080 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.677047    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz" event={"ID":"93adf525-5fd1-4beb-bc15-990bbf213b5c","Type":"ContainerDied","Data":"f37f791d03b5658e560e068e4dd3f8b70d7ae141f7540a0aa9f0993055be5869"}
Nov 03 19:47:49.677521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.677497    4531 generic.go:356] "Generic (PLEG): container finished" podID="91501c4c-a6cc-479b-b660-17a0604c7346" containerID="bdb46d723768ca21340767fcbf00661e9122e8f9cb2d65726b0b042853dac682" exitCode=0
Nov 03 19:47:49.678061 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.678038    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv" event={"ID":"91501c4c-a6cc-479b-b660-17a0604c7346","Type":"ContainerDied","Data":"bdb46d723768ca21340767fcbf00661e9122e8f9cb2d65726b0b042853dac682"}
Nov 03 19:47:49.681567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.681545    4531 generic.go:356] "Generic (PLEG): container finished" podID="38448d63-7dad-48b1-8373-b72e8ede03af" containerID="d931dcf3840fe03d5a9acdd6f7dd5354116f371dbe4619ccad22ecc6ee940cc5" exitCode=0
Nov 03 19:47:49.682820 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.681754    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml" event={"ID":"38448d63-7dad-48b1-8373-b72e8ede03af","Type":"ContainerDied","Data":"d931dcf3840fe03d5a9acdd6f7dd5354116f371dbe4619ccad22ecc6ee940cc5"}
Nov 03 19:47:49.695023 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.694978    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5" podStartSLOduration=2.694964238 podStartE2EDuration="2.694964238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:49.692965982 +0000 UTC m=+982.963089276" watchObservedRunningTime="2026-11-03 19:47:49.694964238 +0000 UTC m=+982.965087532"
Nov 03 19:47:49.721246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.712642    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:49.769344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.769163    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:47:50.006781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.996766    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:50.006781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.999747    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93adf525-5fd1-4beb-bc15-990bbf213b5c-kube-api-access-7k6rm\" (UniqueName: \"kubernetes.io/projected/93adf525-5fd1-4beb-bc15-990bbf213b5c-kube-api-access-7k6rm\") pod \"93adf525-5fd1-4beb-bc15-990bbf213b5c\" (UID: \"93adf525-5fd1-4beb-bc15-990bbf213b5c\") "
Nov 03 19:47:50.006781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:49.999816    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93adf525-5fd1-4beb-bc15-990bbf213b5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93adf525-5fd1-4beb-bc15-990bbf213b5c-plugins-config-volume\") pod \"93adf525-5fd1-4beb-bc15-990bbf213b5c\" (UID: \"93adf525-5fd1-4beb-bc15-990bbf213b5c\") "
Nov 03 19:47:50.006781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.005768    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93adf525-5fd1-4beb-bc15-990bbf213b5c-plugins-config-volume" pod "93adf525-5fd1-4beb-bc15-990bbf213b5c" (UID: "93adf525-5fd1-4beb-bc15-990bbf213b5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:50.025443 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.025371    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:50.027038 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.026999    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93adf525-5fd1-4beb-bc15-990bbf213b5c-kube-api-access-7k6rm" pod "93adf525-5fd1-4beb-bc15-990bbf213b5c" (UID: "93adf525-5fd1-4beb-bc15-990bbf213b5c"). InnerVolumeSpecName "kube-api-access-7k6rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:50.068833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.068332    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:50.070520 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.070490    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:50.103645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.103603    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:50.113006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.112919    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k6rm\" (UniqueName: \"kubernetes.io/projected/93adf525-5fd1-4beb-bc15-990bbf213b5c-kube-api-access-7k6rm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.113321 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.113296    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93adf525-5fd1-4beb-bc15-990bbf213b5c-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.233973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.232197    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38448d63-7dad-48b1-8373-b72e8ede03af-kube-api-access-rhv9b\" (UniqueName: \"kubernetes.io/projected/38448d63-7dad-48b1-8373-b72e8ede03af-kube-api-access-rhv9b\") pod \"38448d63-7dad-48b1-8373-b72e8ede03af\" (UID: \"38448d63-7dad-48b1-8373-b72e8ede03af\") "
Nov 03 19:47:50.233973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.232297    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91501c4c-a6cc-479b-b660-17a0604c7346-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91501c4c-a6cc-479b-b660-17a0604c7346-plugins-config-volume\") pod \"91501c4c-a6cc-479b-b660-17a0604c7346\" (UID: \"91501c4c-a6cc-479b-b660-17a0604c7346\") "
Nov 03 19:47:50.233973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.232330    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38448d63-7dad-48b1-8373-b72e8ede03af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38448d63-7dad-48b1-8373-b72e8ede03af-plugins-config-volume\") pod \"38448d63-7dad-48b1-8373-b72e8ede03af\" (UID: \"38448d63-7dad-48b1-8373-b72e8ede03af\") "
Nov 03 19:47:50.233973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.232612    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91501c4c-a6cc-479b-b660-17a0604c7346-kube-api-access-24642\" (UniqueName: \"kubernetes.io/projected/91501c4c-a6cc-479b-b660-17a0604c7346-kube-api-access-24642\") pod \"91501c4c-a6cc-479b-b660-17a0604c7346\" (UID: \"91501c4c-a6cc-479b-b660-17a0604c7346\") "
Nov 03 19:47:50.239613 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.238242    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38448d63-7dad-48b1-8373-b72e8ede03af-plugins-config-volume" pod "38448d63-7dad-48b1-8373-b72e8ede03af" (UID: "38448d63-7dad-48b1-8373-b72e8ede03af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:50.240624 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.240379    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91501c4c-a6cc-479b-b660-17a0604c7346-plugins-config-volume" pod "91501c4c-a6cc-479b-b660-17a0604c7346" (UID: "91501c4c-a6cc-479b-b660-17a0604c7346"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:50.241933 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.241897    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38448d63-7dad-48b1-8373-b72e8ede03af-kube-api-access-rhv9b" pod "38448d63-7dad-48b1-8373-b72e8ede03af" (UID: "38448d63-7dad-48b1-8373-b72e8ede03af"). InnerVolumeSpecName "kube-api-access-rhv9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:50.242190 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.242140    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91501c4c-a6cc-479b-b660-17a0604c7346-kube-api-access-24642" pod "91501c4c-a6cc-479b-b660-17a0604c7346" (UID: "91501c4c-a6cc-479b-b660-17a0604c7346"). InnerVolumeSpecName "kube-api-access-24642". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:50.266455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.265686    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"]
Nov 03 19:47:50.335480 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.335323    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhv9b\" (UniqueName: \"kubernetes.io/projected/38448d63-7dad-48b1-8373-b72e8ede03af-kube-api-access-rhv9b\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.335480 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.335360    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91501c4c-a6cc-479b-b660-17a0604c7346-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.335480 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.335376    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38448d63-7dad-48b1-8373-b72e8ede03af-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.335480 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.335391    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24642\" (UniqueName: \"kubernetes.io/projected/91501c4c-a6cc-479b-b660-17a0604c7346-kube-api-access-24642\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.394540 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:50.394503    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d1aab5_730f_4393_a956_d721f1a7fcc0.slice/crio-30c210c9c2c68a1b1879b34adcb33fa85ebabf7d2af0acca186494568d2b4913 WatchSource:0}: Error finding container 30c210c9c2c68a1b1879b34adcb33fa85ebabf7d2af0acca186494568d2b4913: Status 404 returned error can't find the container with id 30c210c9c2c68a1b1879b34adcb33fa85ebabf7d2af0acca186494568d2b4913
Nov 03 19:47:50.400835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.400805    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"]
Nov 03 19:47:50.575671 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:47:50.575578    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:47:50.694743 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.694697    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76" event={"ID":"bba12cce-ee41-46d3-a181-b01838b310cd","Type":"ContainerStarted","Data":"4f262ae4dbf4201af38924f279662a58e9474c307bd1ec4461af625b2bc8b445"}
Nov 03 19:47:50.694937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.694773    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76" event={"ID":"bba12cce-ee41-46d3-a181-b01838b310cd","Type":"ContainerStarted","Data":"2a87bbe5d91871ce2540349f8f1ae08b811e0cf6365cb2f4f04237989863b0ae"}
Nov 03 19:47:50.694937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.694803    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:47:50.696576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.696552    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"
Nov 03 19:47:50.696708 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.696597    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml" event={"ID":"38448d63-7dad-48b1-8373-b72e8ede03af","Type":"ContainerDied","Data":"5a8a2ab7f8dc9360af7b518c4b698254a00cda873803df32cff94c708e52099f"}
Nov 03 19:47:50.696708 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.696658    4531 scope.go:122] "RemoveContainer" containerID="d931dcf3840fe03d5a9acdd6f7dd5354116f371dbe4619ccad22ecc6ee940cc5"
Nov 03 19:47:50.698685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.698660    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm" event={"ID":"f7d1aab5-730f-4393-a956-d721f1a7fcc0","Type":"ContainerStarted","Data":"13a40da5e52e2ce9843235fa0413a05e43b8046d6e8edf084d65c064bc3d8f9c"}
Nov 03 19:47:50.698826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.698698    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm" event={"ID":"f7d1aab5-730f-4393-a956-d721f1a7fcc0","Type":"ContainerStarted","Data":"30c210c9c2c68a1b1879b34adcb33fa85ebabf7d2af0acca186494568d2b4913"}
Nov 03 19:47:50.699379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.699358    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:47:50.700959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.700932    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"
Nov 03 19:47:50.701069 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.700952    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz" event={"ID":"93adf525-5fd1-4beb-bc15-990bbf213b5c","Type":"ContainerDied","Data":"279c02ad9031472d571d7ee0e224088946549bfff4ba8ddc8aaf776c92daeb90"}
Nov 03 19:47:50.702553 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.702515    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"
Nov 03 19:47:50.702645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.702547    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv" event={"ID":"91501c4c-a6cc-479b-b660-17a0604c7346","Type":"ContainerDied","Data":"ffb9074fcf4ed0d20091eec0c56e86e6766c2542f1a3e1cd506405fea416edb5"}
Nov 03 19:47:50.703015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.702962    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:50.705036 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.705018    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:47:50.715331 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.715284    4531 scope.go:122] "RemoveContainer" containerID="f37f791d03b5658e560e068e4dd3f8b70d7ae141f7540a0aa9f0993055be5869"
Nov 03 19:47:50.717354 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.717306    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76" podStartSLOduration=2.717289985 podStartE2EDuration="2.717289985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:50.708461684 +0000 UTC m=+983.978584980" watchObservedRunningTime="2026-11-03 19:47:50.717289985 +0000 UTC m=+983.987413281"
Nov 03 19:47:50.726905 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.726880    4531 scope.go:122] "RemoveContainer" containerID="bdb46d723768ca21340767fcbf00661e9122e8f9cb2d65726b0b042853dac682"
Nov 03 19:47:50.749433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.749370    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm" podStartSLOduration=2.749350818 podStartE2EDuration="2.749350818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:50.744223422 +0000 UTC m=+984.014346718" watchObservedRunningTime="2026-11-03 19:47:50.749350818 +0000 UTC m=+984.019474113"
Nov 03 19:47:50.783584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.783553    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"]
Nov 03 19:47:50.791671 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.791635    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e17271b-67fb547655-8lkcz"]
Nov 03 19:47:50.798641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.798607    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"]
Nov 03 19:47:50.806167 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.806116    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e17271b-769fd8b889-gcnml"]
Nov 03 19:47:50.812926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.812889    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"]
Nov 03 19:47:50.818892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.818867    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cbfcf47a-797d9f5d99-j2nzv"]
Nov 03 19:47:50.952470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.952428    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38448d63-7dad-48b1-8373-b72e8ede03af" path="/var/lib/kubelet/pods/38448d63-7dad-48b1-8373-b72e8ede03af/volumes"
Nov 03 19:47:50.953719 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.953677    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91501c4c-a6cc-479b-b660-17a0604c7346" path="/var/lib/kubelet/pods/91501c4c-a6cc-479b-b660-17a0604c7346/volumes"
Nov 03 19:47:50.954920 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:50.954887    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93adf525-5fd1-4beb-bc15-990bbf213b5c" path="/var/lib/kubelet/pods/93adf525-5fd1-4beb-bc15-990bbf213b5c/volumes"
Nov 03 19:47:51.379066 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:51.379025    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:51.728464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:51.728383    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:51.879780 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:51.879748    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:52.044250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:52.037740    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:52.070213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:52.068365    4531 patch_prober.go:29] interesting pod/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:52.070213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:52.068431    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr" podUID="7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:52.239201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:52.239135    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:52.455199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:52.454010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:53.044005 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.043857    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:53.054654 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.054607    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"]
Nov 03 19:47:53.055819 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.055777    4531 cpu_manager.go:414] "containerMap: removing container" podUID="38448d63-7dad-48b1-8373-b72e8ede03af" containerName="payload-processing"
Nov 03 19:47:53.055819 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.055804    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38448d63-7dad-48b1-8373-b72e8ede03af" containerName="payload-processing"
Nov 03 19:47:53.056082 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.055856    4531 cpu_manager.go:414] "containerMap: removing container" podUID="93adf525-5fd1-4beb-bc15-990bbf213b5c" containerName="payload-pre-processing"
Nov 03 19:47:53.056082 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.055864    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93adf525-5fd1-4beb-bc15-990bbf213b5c" containerName="payload-pre-processing"
Nov 03 19:47:53.056082 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.055913    4531 cpu_manager.go:414] "containerMap: removing container" podUID="91501c4c-a6cc-479b-b660-17a0604c7346" containerName="payload-processing"
Nov 03 19:47:53.056082 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.055921    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91501c4c-a6cc-479b-b660-17a0604c7346" containerName="payload-processing"
Nov 03 19:47:53.056431 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.056097    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="91501c4c-a6cc-479b-b660-17a0604c7346" containerName="payload-processing"
Nov 03 19:47:53.056431 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.056113    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="38448d63-7dad-48b1-8373-b72e8ede03af" containerName="payload-processing"
Nov 03 19:47:53.056431 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.056130    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="93adf525-5fd1-4beb-bc15-990bbf213b5c" containerName="payload-pre-processing"
Nov 03 19:47:53.073685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.072655    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:47:53.073896 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.072558    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"]
Nov 03 19:47:53.080708 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.080116    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b3a1058c\""
Nov 03 19:47:53.080708 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.080302    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"]
Nov 03 19:47:53.085784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.085756    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:47:53.092832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.092807    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b3a1058c-dockercfg-99zgd\""
Nov 03 19:47:53.094256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.094230    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"]
Nov 03 19:47:53.170488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.170455    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:53.180904 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.180871    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:53.204630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.204593    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a546ae-c169-464c-b16c-8667bd96879b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn\" (UID: \"c4a546ae-c169-464c-b16c-8667bd96879b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:47:53.204782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.204640    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sncdr\" (UniqueName: \"kubernetes.io/projected/c4a546ae-c169-464c-b16c-8667bd96879b-kube-api-access-sncdr\") pod \"payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn\" (UID: \"c4a546ae-c169-464c-b16c-8667bd96879b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:47:53.204840 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.204788    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77d22\" (UniqueName: \"kubernetes.io/projected/0b22a5cc-b303-4db6-b943-61473387c8b4-kube-api-access-77d22\") pod \"payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth\" (UID: \"0b22a5cc-b303-4db6-b943-61473387c8b4\") " pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:47:53.205069 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.205045    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b22a5cc-b303-4db6-b943-61473387c8b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth\" (UID: \"0b22a5cc-b303-4db6-b943-61473387c8b4\") " pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:47:53.314196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.313677    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-socket\") pod \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") "
Nov 03 19:47:53.314196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.313792    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-podinfo\") pod \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") "
Nov 03 19:47:53.314196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.313891    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-token\" (UniqueName: \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-token\") pod \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") "
Nov 03 19:47:53.314196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.313931    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-credential-socket\") pod \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") "
Nov 03 19:47:53.314196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.314198    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-certs\") pod \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") "
Nov 03 19:47:53.314608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.314254    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-envoy\") pod \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") "
Nov 03 19:47:53.314608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.314327    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-data\") pod \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") "
Nov 03 19:47:53.314608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.314373    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istiod-ca-cert\") pod \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") "
Nov 03 19:47:53.314608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.314404    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-kube-api-access-8dnsl\" (UniqueName: \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-kube-api-access-8dnsl\") pod \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\" (UID: \"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8\") "
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.314800    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b22a5cc-b303-4db6-b943-61473387c8b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth\" (UID: \"0b22a5cc-b303-4db6-b943-61473387c8b4\") " pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.315043    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a546ae-c169-464c-b16c-8667bd96879b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn\" (UID: \"c4a546ae-c169-464c-b16c-8667bd96879b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.315080    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sncdr\" (UniqueName: \"kubernetes.io/projected/c4a546ae-c169-464c-b16c-8667bd96879b-kube-api-access-sncdr\") pod \"payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn\" (UID: \"c4a546ae-c169-464c-b16c-8667bd96879b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.315220    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77d22\" (UniqueName: \"kubernetes.io/projected/0b22a5cc-b303-4db6-b943-61473387c8b4-kube-api-access-77d22\") pod \"payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth\" (UID: \"0b22a5cc-b303-4db6-b943-61473387c8b4\") " pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.339912    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-data" pod "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" (UID: "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.342233    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-credential-socket" pod "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" (UID: "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.344337    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-socket" pod "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" (UID: "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.353437    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.358053    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a546ae-c169-464c-b16c-8667bd96879b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn\" (UID: \"c4a546ae-c169-464c-b16c-8667bd96879b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.363752    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-certs" pod "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" (UID: "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.364484    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b22a5cc-b303-4db6-b943-61473387c8b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth\" (UID: \"0b22a5cc-b303-4db6-b943-61473387c8b4\") " pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.380506    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istiod-ca-cert" pod "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" (UID: "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.395393    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-podinfo" pod "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" (UID: "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:47:53.410301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.395417    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-token" pod "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" (UID: "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:53.428737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.421523    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.428737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.421555    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.428737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.421571    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.428737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.421635    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.428737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.421649    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.428737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.421660    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.428737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.421674    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.431264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.430983    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-kube-api-access-8dnsl" pod "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" (UID: "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8"). InnerVolumeSpecName "kube-api-access-8dnsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:53.456234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.453957    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sncdr\" (UniqueName: \"kubernetes.io/projected/c4a546ae-c169-464c-b16c-8667bd96879b-kube-api-access-sncdr\") pod \"payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn\" (UID: \"c4a546ae-c169-464c-b16c-8667bd96879b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:47:53.456234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.455402    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77d22\" (UniqueName: \"kubernetes.io/projected/0b22a5cc-b303-4db6-b943-61473387c8b4-kube-api-access-77d22\") pod \"payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth\" (UID: \"0b22a5cc-b303-4db6-b943-61473387c8b4\") " pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:47:53.466196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.463384    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-envoy" pod "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" (UID: "7e5daacc-8bb6-45e1-b56a-4b5ba88edac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:53.527196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.523864    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.527196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.523920    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dnsl\" (UniqueName: \"kubernetes.io/projected/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8-kube-api-access-8dnsl\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.724596 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.724515    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:47:53.729455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.728430    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:47:53.836057 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.836020    4531 generic.go:356] "Generic (PLEG): container finished" podID="7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" containerID="18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719" exitCode=0
Nov 03 19:47:53.837687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.836336    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"
Nov 03 19:47:53.837687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.836351    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr" event={"ID":"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8","Type":"ContainerDied","Data":"18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719"}
Nov 03 19:47:53.837687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.836445    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr" event={"ID":"7e5daacc-8bb6-45e1-b56a-4b5ba88edac8","Type":"ContainerDied","Data":"61d5f1e89d1b1964980f1155104d65cb8f1cff70f32b2031a81cbd96a51f1ac8"}
Nov 03 19:47:53.837687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.836468    4531 scope.go:122] "RemoveContainer" containerID="18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719"
Nov 03 19:47:53.858560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.858537    4531 scope.go:122] "RemoveContainer" containerID="18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719"
Nov 03 19:47:53.859123 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:47:53.859052    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719\": container with ID starting with 18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719 not found: ID does not exist" containerID="18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719"
Nov 03 19:47:53.859123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.859090    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719"} err="failed to get container status \"18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719\": rpc error: code = NotFound desc = could not find container \"18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719\": container with ID starting with 18a88a15de3e80ab0aae890a0675179a4730c1a9f04c1c77e97eaeb7ec428719 not found: ID does not exist"
Nov 03 19:47:53.874835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.874805    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"]
Nov 03 19:47:53.880883 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:53.880843    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-56nxr"]
Nov 03 19:47:54.056115 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.056056    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"]
Nov 03 19:47:54.130377 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.124427    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:54.168045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.168020    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"]
Nov 03 19:47:54.848968 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.848937    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn" event={"ID":"c4a546ae-c169-464c-b16c-8667bd96879b","Type":"ContainerStarted","Data":"1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd"}
Nov 03 19:47:54.849182 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.848995    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn" event={"ID":"c4a546ae-c169-464c-b16c-8667bd96879b","Type":"ContainerStarted","Data":"ea01029c8efa09e953fbae85c2d4173c93cc3d0877b2358913456a2062bdf0a2"}
Nov 03 19:47:54.849182 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.849022    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:47:54.860608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.860580    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" path="/var/lib/kubelet/pods/7e5daacc-8bb6-45e1-b56a-4b5ba88edac8/volumes"
Nov 03 19:47:54.860957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.860944    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:47:54.861015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.860986    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth" event={"ID":"0b22a5cc-b303-4db6-b943-61473387c8b4","Type":"ContainerStarted","Data":"940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25"}
Nov 03 19:47:54.861057 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.861037    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth" event={"ID":"0b22a5cc-b303-4db6-b943-61473387c8b4","Type":"ContainerStarted","Data":"8ab426db2157f3e7a6ff37a5e9ad7aea9e1c09867caebfa06e15cc06db04cc31"}
Nov 03 19:47:54.867862 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.867778    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn" podStartSLOduration=1.867762669 podStartE2EDuration="1.867762669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:54.862313835 +0000 UTC m=+988.132437131" watchObservedRunningTime="2026-11-03 19:47:54.867762669 +0000 UTC m=+988.137885968"
Nov 03 19:47:54.881417 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.881365    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth" podStartSLOduration=1.88135059 podStartE2EDuration="1.88135059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:54.877389911 +0000 UTC m=+988.147513203" watchObservedRunningTime="2026-11-03 19:47:54.88135059 +0000 UTC m=+988.151473885"
Nov 03 19:47:54.923513 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:54.923482    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:55.658398 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:55.658359    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:56.470199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:56.466847    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:57.137861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.137818    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:57.743506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.743466    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"]
Nov 03 19:47:57.745452 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.745375    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" containerName="istio-proxy"
Nov 03 19:47:57.745584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.745457    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" containerName="istio-proxy"
Nov 03 19:47:57.759493 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.759203    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5daacc-8bb6-45e1-b56a-4b5ba88edac8" containerName="istio-proxy"
Nov 03 19:47:57.763907 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.763854    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"]
Nov 03 19:47:57.764031 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.763937    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:57.768266 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.768241    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c96ee058-serving-cert\""
Nov 03 19:47:57.826590 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.826549    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:57.867886 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.867853    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-maas-api-tls\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:57.868052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.868035    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f5b246-ac88-4b7d-a616-093d1adf36c5-openshift-service-ca\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:57.868100 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.868084    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-metrics-certs\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:57.868241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.868227    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k48b\" (UniqueName: \"kubernetes.io/projected/52f5b246-ac88-4b7d-a616-093d1adf36c5-kube-api-access-5k48b\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:57.976195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.971720    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f5b246-ac88-4b7d-a616-093d1adf36c5-openshift-service-ca\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:57.976195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.971820    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-metrics-certs\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:57.976195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.971999    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k48b\" (UniqueName: \"kubernetes.io/projected/52f5b246-ac88-4b7d-a616-093d1adf36c5-kube-api-access-5k48b\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:57.976195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.972140    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-maas-api-tls\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:57.976195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.974683    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f5b246-ac88-4b7d-a616-093d1adf36c5-openshift-service-ca\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:58.004244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:57.994513    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-maas-api-tls\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:58.023203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:58.012890    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k48b\" (UniqueName: \"kubernetes.io/projected/52f5b246-ac88-4b7d-a616-093d1adf36c5-kube-api-access-5k48b\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:58.035486 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:58.035451    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:58.037751 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:58.036336    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-metrics-certs\") pod \"maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:58.081288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:58.081002    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:47:58.501963 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:58.501924    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"]
Nov 03 19:47:58.515071 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:47:58.515003    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f5b246_ac88_4b7d_a616_093d1adf36c5.slice/crio-266c70991d77f1297e75036abc91c262f7250aa1af329f7fa0eb7284a8ac1e6a WatchSource:0}: Error finding container 266c70991d77f1297e75036abc91c262f7250aa1af329f7fa0eb7284a8ac1e6a: Status 404 returned error can't find the container with id 266c70991d77f1297e75036abc91c262f7250aa1af329f7fa0eb7284a8ac1e6a
Nov 03 19:47:58.666197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:58.665450    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:59.044256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.041790    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb" event={"ID":"52f5b246-ac88-4b7d-a616-093d1adf36c5","Type":"ContainerStarted","Data":"266c70991d77f1297e75036abc91c262f7250aa1af329f7fa0eb7284a8ac1e6a"}
Nov 03 19:47:59.333882 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.333807    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"]
Nov 03 19:47:59.341695 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.341625    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.351349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.351321    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b3a1058c-serving-cert\""
Nov 03 19:47:59.359470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.359442    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"]
Nov 03 19:47:59.451924 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.451885    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgtg2\" (UniqueName: \"kubernetes.io/projected/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-kube-api-access-zgtg2\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.452096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.452038    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-metrics-certs\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.452234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.452186    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-maas-api-tls\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.452577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.452444    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-openshift-service-ca\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.477904 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.477864    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:59.553811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.553775    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-openshift-service-ca\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.554285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.553892    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgtg2\" (UniqueName: \"kubernetes.io/projected/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-kube-api-access-zgtg2\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.554285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.554004    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-metrics-certs\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.554285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.554125    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-maas-api-tls\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.554600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.554573    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-openshift-service-ca\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.556712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.556695    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-maas-api-tls\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.556791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.556756    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-metrics-certs\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.561343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.561319    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgtg2\" (UniqueName: \"kubernetes.io/projected/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-kube-api-access-zgtg2\") pod \"maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:47:59.633968 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.633930    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:59.674835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:47:59.672395    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:48:00.027213 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:48:00.026151    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f42caee_4e6c_4d70_b9b3_18490efc7ea1.slice/crio-1f37f4c794571934277cd762a2d3d392908ebc581fcc5c48bc3158c43ffdf2c3 WatchSource:0}: Error finding container 1f37f4c794571934277cd762a2d3d392908ebc581fcc5c48bc3158c43ffdf2c3: Status 404 returned error can't find the container with id 1f37f4c794571934277cd762a2d3d392908ebc581fcc5c48bc3158c43ffdf2c3
Nov 03 19:48:00.028913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:00.028887    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"]
Nov 03 19:48:00.047188 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:00.047147    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:00.053745 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:00.053711    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb" event={"ID":"52f5b246-ac88-4b7d-a616-093d1adf36c5","Type":"ContainerStarted","Data":"b4a2040aa319461de24759e466a3dcb99e7dbcb902bb01f1b21bdc10f10435da"}
Nov 03 19:48:00.053874 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:00.053785    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:48:00.055222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:00.055196    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t" event={"ID":"3f42caee-4e6c-4d70-b9b3-18490efc7ea1","Type":"ContainerStarted","Data":"1f37f4c794571934277cd762a2d3d392908ebc581fcc5c48bc3158c43ffdf2c3"}
Nov 03 19:48:00.075290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:00.075225    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb" podStartSLOduration=2.53889488 podStartE2EDuration="3.075211331s" totalImagesPullingTime="536.316451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:00.067044948 +0000 UTC m=+993.337168244" watchObservedRunningTime="2026-11-03 19:48:00.075211331 +0000 UTC m=+993.345334626"
Nov 03 19:48:00.301254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:00.300241    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"]
Nov 03 19:48:00.320197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:00.316237    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26c00c22-xc7rv"]
Nov 03 19:48:00.511196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:00.507762    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:00.736382 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:00.735794    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:01.068604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:01.068269    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f7cd946-a247-447b-8812-4b31be2f12d5" path="/var/lib/kubelet/pods/0f7cd946-a247-447b-8812-4b31be2f12d5/volumes"
Nov 03 19:48:01.420651 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:01.420614    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:01.593200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:01.590863    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:01.742303 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:01.741036    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:48:01.769196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:01.764580    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:02.126947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:02.126861    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t" event={"ID":"3f42caee-4e6c-4d70-b9b3-18490efc7ea1","Type":"ContainerStarted","Data":"ec222eafe1649741cf21708519e48d7dc552a9cb90e27c5800959e38424cbc8d"}
Nov 03 19:48:02.127119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:02.126995    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:48:02.147812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:02.147759    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t" podStartSLOduration=2.440681913 podStartE2EDuration="3.14774539s" totalImagesPullingTime="707.063477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:02.143889357 +0000 UTC m=+995.414012652" watchObservedRunningTime="2026-11-03 19:48:02.14774539 +0000 UTC m=+995.417868683"
Nov 03 19:48:02.224396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:02.224356    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:02.385106 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:02.385002    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:02.816000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:02.815926    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:48:02.968044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:02.968011    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:03.699195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:03.691094    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:04.374198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:04.370284    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:05.152198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:05.141327    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:05.838195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:05.836030    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:05.870070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:05.870042    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:48:05.875706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:05.875660    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:48:06.121968 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:06.121888    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:48:06.503716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:06.503680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:07.359665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:07.359615    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:07.929494 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:07.929460    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:08.142433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:08.142403    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:48:08.790587 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:08.790556    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:09.613378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:09.613339    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:10.735194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:10.734703    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:12.264831 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:12.264775    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:12.464286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:12.446282    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:48:12.489387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:12.489346    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:48:12.489908 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:12.489879    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:48:12.913251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:12.913202    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:13.085212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:13.085143    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:13.242254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:13.233341    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:13.259199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:13.251609    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:13.500252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:13.491284    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:13.500252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:13.491984    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:48:13.831348 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:13.830356    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:48:14.197197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:14.188865    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:14.252682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:14.252639    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:14.912311 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:14.802262    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded"
Nov 03 19:48:14.912311 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:14.802730    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:14.912311 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:14.833280    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:15.065193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.064475    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 03 19:48:15.113197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.090402    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:15.136107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.113692    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:48:15.211794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.210703    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:15.257362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.257306    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:15.341412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.341308    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:15.436211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.430119    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:48:15.436211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.432081    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:48:15.436211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.434343    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:48:15.500099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.499898    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:15.632216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.632185    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:48:15.964226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:15.964192    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:16.662308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:16.655666    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:17.284200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:17.284152    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:17.936857 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:17.936818    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:18.678198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:18.677986    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:19.069414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:19.069330    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"]
Nov 03 19:48:19.083351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:19.083319    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"]
Nov 03 19:48:19.179523 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:19.179487    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:19.307331 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:19.307298    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:19.747844 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:19.747808    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:20.567200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:20.567114    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:20.658402 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:48:20.658364    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:48:21.167576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:21.167539    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:21.735113 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:21.735075    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:22.476002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:22.475965    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:23.583736 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:23.583705    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:24.070305 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:24.070260    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76" podUID="bba12cce-ee41-46d3-a181-b01838b310cd" containerName="payload-processing" containerID="cri-o://4f262ae4dbf4201af38924f279662a58e9474c307bd1ec4461af625b2bc8b445" gracePeriod=25
Nov 03 19:48:24.083719 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:24.083678    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm" podUID="f7d1aab5-730f-4393-a956-d721f1a7fcc0" containerName="payload-pre-processing" containerID="cri-o://13a40da5e52e2ce9843235fa0413a05e43b8046d6e8edf084d65c064bc3d8f9c" gracePeriod=25
Nov 03 19:48:24.111789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:24.111757    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"]
Nov 03 19:48:24.112070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:24.112040    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb" podUID="52f5b246-ac88-4b7d-a616-093d1adf36c5" containerName="maas-api" containerID="cri-o://b4a2040aa319461de24759e466a3dcb99e7dbcb902bb01f1b21bdc10f10435da" gracePeriod=30
Nov 03 19:48:24.189318 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:24.189285    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:24.345246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:24.339846    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:24.495203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:24.493391    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:24.679889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:24.667423    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:25.005346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.003019    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:25.176204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.176144    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"]
Nov 03 19:48:25.183592 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.176394    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7" podUID="24a7070e-597a-4d4f-9f62-ea93932fffd3" containerName="istio-proxy" containerID="cri-o://9facc79e9d4af4a75b881d5f163186f5f41507504ca4b69009e4b8d6d78a49f3" gracePeriod=30
Nov 03 19:48:25.228021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.227981    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm" event={"ID":"f7d1aab5-730f-4393-a956-d721f1a7fcc0","Type":"ContainerDied","Data":"13a40da5e52e2ce9843235fa0413a05e43b8046d6e8edf084d65c064bc3d8f9c"}
Nov 03 19:48:25.228021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.228010    4531 generic.go:356] "Generic (PLEG): container finished" podID="f7d1aab5-730f-4393-a956-d721f1a7fcc0" containerID="13a40da5e52e2ce9843235fa0413a05e43b8046d6e8edf084d65c064bc3d8f9c" exitCode=0
Nov 03 19:48:25.228555 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.228531    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:48:25.230141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.230117    4531 generic.go:356] "Generic (PLEG): container finished" podID="52f5b246-ac88-4b7d-a616-093d1adf36c5" containerID="b4a2040aa319461de24759e466a3dcb99e7dbcb902bb01f1b21bdc10f10435da" exitCode=0
Nov 03 19:48:25.230534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.230512    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb" event={"ID":"52f5b246-ac88-4b7d-a616-093d1adf36c5","Type":"ContainerDied","Data":"b4a2040aa319461de24759e466a3dcb99e7dbcb902bb01f1b21bdc10f10435da"}
Nov 03 19:48:25.230697 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.230682    4531 scope.go:122] "RemoveContainer" containerID="b4a2040aa319461de24759e466a3dcb99e7dbcb902bb01f1b21bdc10f10435da"
Nov 03 19:48:25.234323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.234302    4531 generic.go:356] "Generic (PLEG): container finished" podID="bba12cce-ee41-46d3-a181-b01838b310cd" containerID="4f262ae4dbf4201af38924f279662a58e9474c307bd1ec4461af625b2bc8b445" exitCode=0
Nov 03 19:48:25.234605 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.234589    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76" event={"ID":"bba12cce-ee41-46d3-a181-b01838b310cd","Type":"ContainerDied","Data":"4f262ae4dbf4201af38924f279662a58e9474c307bd1ec4461af625b2bc8b445"}
Nov 03 19:48:25.264245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.264139    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-metrics-certs\") pod \"52f5b246-ac88-4b7d-a616-093d1adf36c5\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") "
Nov 03 19:48:25.264386 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.264287    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f5b246-ac88-4b7d-a616-093d1adf36c5-kube-api-access-5k48b\" (UniqueName: \"kubernetes.io/projected/52f5b246-ac88-4b7d-a616-093d1adf36c5-kube-api-access-5k48b\") pod \"52f5b246-ac88-4b7d-a616-093d1adf36c5\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") "
Nov 03 19:48:25.264457 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.264406    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-maas-api-tls\") pod \"52f5b246-ac88-4b7d-a616-093d1adf36c5\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") "
Nov 03 19:48:25.264597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.264577    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f5b246-ac88-4b7d-a616-093d1adf36c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f5b246-ac88-4b7d-a616-093d1adf36c5-openshift-service-ca\") pod \"52f5b246-ac88-4b7d-a616-093d1adf36c5\" (UID: \"52f5b246-ac88-4b7d-a616-093d1adf36c5\") "
Nov 03 19:48:25.265747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.265710    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f5b246-ac88-4b7d-a616-093d1adf36c5-openshift-service-ca" pod "52f5b246-ac88-4b7d-a616-093d1adf36c5" (UID: "52f5b246-ac88-4b7d-a616-093d1adf36c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:25.267215 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.267187    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-metrics-certs" pod "52f5b246-ac88-4b7d-a616-093d1adf36c5" (UID: "52f5b246-ac88-4b7d-a616-093d1adf36c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:25.267568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.267531    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-maas-api-tls" pod "52f5b246-ac88-4b7d-a616-093d1adf36c5" (UID: "52f5b246-ac88-4b7d-a616-093d1adf36c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:25.268431 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.268406    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f5b246-ac88-4b7d-a616-093d1adf36c5-kube-api-access-5k48b" pod "52f5b246-ac88-4b7d-a616-093d1adf36c5" (UID: "52f5b246-ac88-4b7d-a616-093d1adf36c5"). InnerVolumeSpecName "kube-api-access-5k48b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:25.273785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.273763    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:48:25.277116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.277092    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:48:25.370235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.367360    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d1aab5-730f-4393-a956-d721f1a7fcc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d1aab5-730f-4393-a956-d721f1a7fcc0-plugins-config-volume\") pod \"f7d1aab5-730f-4393-a956-d721f1a7fcc0\" (UID: \"f7d1aab5-730f-4393-a956-d721f1a7fcc0\") "
Nov 03 19:48:25.370235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.367469    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d1aab5-730f-4393-a956-d721f1a7fcc0-kube-api-access-58lsv\" (UniqueName: \"kubernetes.io/projected/f7d1aab5-730f-4393-a956-d721f1a7fcc0-kube-api-access-58lsv\") pod \"f7d1aab5-730f-4393-a956-d721f1a7fcc0\" (UID: \"f7d1aab5-730f-4393-a956-d721f1a7fcc0\") "
Nov 03 19:48:25.370235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.367520    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba12cce-ee41-46d3-a181-b01838b310cd-kube-api-access-ghwf6\" (UniqueName: \"kubernetes.io/projected/bba12cce-ee41-46d3-a181-b01838b310cd-kube-api-access-ghwf6\") pod \"bba12cce-ee41-46d3-a181-b01838b310cd\" (UID: \"bba12cce-ee41-46d3-a181-b01838b310cd\") "
Nov 03 19:48:25.370235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.368788    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba12cce-ee41-46d3-a181-b01838b310cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba12cce-ee41-46d3-a181-b01838b310cd-plugins-config-volume\") pod \"bba12cce-ee41-46d3-a181-b01838b310cd\" (UID: \"bba12cce-ee41-46d3-a181-b01838b310cd\") "
Nov 03 19:48:25.370235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.369725    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f5b246-ac88-4b7d-a616-093d1adf36c5-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:25.370235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.369749    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:25.370235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.369763    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k48b\" (UniqueName: \"kubernetes.io/projected/52f5b246-ac88-4b7d-a616-093d1adf36c5-kube-api-access-5k48b\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:25.370235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.369778    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f5b246-ac88-4b7d-a616-093d1adf36c5-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:25.370235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.370135    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba12cce-ee41-46d3-a181-b01838b310cd-plugins-config-volume" pod "bba12cce-ee41-46d3-a181-b01838b310cd" (UID: "bba12cce-ee41-46d3-a181-b01838b310cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:25.370235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.370163    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d1aab5-730f-4393-a956-d721f1a7fcc0-plugins-config-volume" pod "f7d1aab5-730f-4393-a956-d721f1a7fcc0" (UID: "f7d1aab5-730f-4393-a956-d721f1a7fcc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:25.373838 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.373804    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d1aab5-730f-4393-a956-d721f1a7fcc0-kube-api-access-58lsv" pod "f7d1aab5-730f-4393-a956-d721f1a7fcc0" (UID: "f7d1aab5-730f-4393-a956-d721f1a7fcc0"). InnerVolumeSpecName "kube-api-access-58lsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:25.374544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.374525    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba12cce-ee41-46d3-a181-b01838b310cd-kube-api-access-ghwf6" pod "bba12cce-ee41-46d3-a181-b01838b310cd" (UID: "bba12cce-ee41-46d3-a181-b01838b310cd"). InnerVolumeSpecName "kube-api-access-ghwf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:25.423279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.423230    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:25.471456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.471425    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d1aab5-730f-4393-a956-d721f1a7fcc0-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:25.471456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.471450    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58lsv\" (UniqueName: \"kubernetes.io/projected/f7d1aab5-730f-4393-a956-d721f1a7fcc0-kube-api-access-58lsv\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:25.471456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.471464    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghwf6\" (UniqueName: \"kubernetes.io/projected/bba12cce-ee41-46d3-a181-b01838b310cd-kube-api-access-ghwf6\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:25.471734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.471472    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba12cce-ee41-46d3-a181-b01838b310cd-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:25.569243 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.563712    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:25.735490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.735439    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"]
Nov 03 19:48:25.736450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.736428    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f7d1aab5-730f-4393-a956-d721f1a7fcc0" containerName="payload-pre-processing"
Nov 03 19:48:25.736450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.736449    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d1aab5-730f-4393-a956-d721f1a7fcc0" containerName="payload-pre-processing"
Nov 03 19:48:25.736627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.736480    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bba12cce-ee41-46d3-a181-b01838b310cd" containerName="payload-processing"
Nov 03 19:48:25.736627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.736487    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba12cce-ee41-46d3-a181-b01838b310cd" containerName="payload-processing"
Nov 03 19:48:25.736627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.736503    4531 cpu_manager.go:414] "containerMap: removing container" podUID="52f5b246-ac88-4b7d-a616-093d1adf36c5" containerName="maas-api"
Nov 03 19:48:25.736627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.736509    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f5b246-ac88-4b7d-a616-093d1adf36c5" containerName="maas-api"
Nov 03 19:48:25.736824 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.736749    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d1aab5-730f-4393-a956-d721f1a7fcc0" containerName="payload-pre-processing"
Nov 03 19:48:25.736824 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.736768    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba12cce-ee41-46d3-a181-b01838b310cd" containerName="payload-processing"
Nov 03 19:48:25.736824 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.736781    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f5b246-ac88-4b7d-a616-093d1adf36c5" containerName="maas-api"
Nov 03 19:48:25.741434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.741405    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.748612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.748586    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"]
Nov 03 19:48:25.779548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.779508    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-6eb768-openshift-default-dockercfg-hk5tx\""
Nov 03 19:48:25.783323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.783264    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63cceab-6dee-4689-8767-9a472d99b541-istiod-ca-cert\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.783470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.783329    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qcm8\" (UniqueName: \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-kube-api-access-6qcm8\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.783470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.783435    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63cceab-6dee-4689-8767-9a472d99b541-istio-podinfo\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.783644 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.783628    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-certs\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.783742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.783729    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-socket\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.783863 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.783849    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-credential-socket\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.783924 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.783877    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-envoy\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.783970 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.783929    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-istio-token\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.784030 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.783987    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-data\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.825821 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.825735    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:25.885735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.885699    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-certs\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.886758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.886730    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-socket\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.887879 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.887855    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-credential-socket\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.889078 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.889059    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-envoy\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.889633 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.889612    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-istio-token\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.890311 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.890291    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-data\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.890898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.890876    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63cceab-6dee-4689-8767-9a472d99b541-istiod-ca-cert\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.891680 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.891659    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qcm8\" (UniqueName: \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-kube-api-access-6qcm8\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.892108 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.892087    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63cceab-6dee-4689-8767-9a472d99b541-istio-podinfo\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.892848 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.890678    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-data\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.893007 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.886379    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-certs\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.893119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.887524    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-socket\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.893258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.888859    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-credential-socket\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.893697 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.893675    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63cceab-6dee-4689-8767-9a472d99b541-istiod-ca-cert\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.942196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.938392    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63cceab-6dee-4689-8767-9a472d99b541-istio-podinfo\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.962212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.955569    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-envoy\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:25.965119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.965017    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:25.990642 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.989053    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qcm8\" (UniqueName: \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-kube-api-access-6qcm8\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:26.003505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:25.997687    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-istio-token\") pod \"e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") " pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:26.073197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.072593    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:26.283286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.269966    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm" event={"ID":"f7d1aab5-730f-4393-a956-d721f1a7fcc0","Type":"ContainerDied","Data":"30c210c9c2c68a1b1879b34adcb33fa85ebabf7d2af0acca186494568d2b4913"}
Nov 03 19:48:26.283286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.270051    4531 scope.go:122] "RemoveContainer" containerID="13a40da5e52e2ce9843235fa0413a05e43b8046d6e8edf084d65c064bc3d8f9c"
Nov 03 19:48:26.283286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.270300    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"
Nov 03 19:48:26.312337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.312147    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"
Nov 03 19:48:26.312337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.312244    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb" event={"ID":"52f5b246-ac88-4b7d-a616-093d1adf36c5","Type":"ContainerDied","Data":"266c70991d77f1297e75036abc91c262f7250aa1af329f7fa0eb7284a8ac1e6a"}
Nov 03 19:48:26.314578 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.314556    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76" event={"ID":"bba12cce-ee41-46d3-a181-b01838b310cd","Type":"ContainerDied","Data":"2a87bbe5d91871ce2540349f8f1ae08b811e0cf6365cb2f4f04237989863b0ae"}
Nov 03 19:48:26.314811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.314787    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"
Nov 03 19:48:26.327541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.327493    4531 scope.go:122] "RemoveContainer" containerID="4f262ae4dbf4201af38924f279662a58e9474c307bd1ec4461af625b2bc8b445"
Nov 03 19:48:26.335253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.335226    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"]
Nov 03 19:48:26.348268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.346075    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c96ee058-55bc8f8dcf-xzbgm"]
Nov 03 19:48:26.360281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.360258    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"]
Nov 03 19:48:26.378911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.371245    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c96ee058-5dfbd4dbbc-dbc76"]
Nov 03 19:48:26.378911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.378643    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"]
Nov 03 19:48:26.386143 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.386116    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c96ee058-68c65c7b47-nz2rb"]
Nov 03 19:48:26.481316 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.481287    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:26.624383 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:48:26.624346    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63cceab_6dee_4689_8767_9a472d99b541.slice/crio-b3c5cc652b35324bc46c14285dfe8888567de74ac6e556eb0c90cda5b3a21306 WatchSource:0}: Error finding container b3c5cc652b35324bc46c14285dfe8888567de74ac6e556eb0c90cda5b3a21306: Status 404 returned error can't find the container with id b3c5cc652b35324bc46c14285dfe8888567de74ac6e556eb0c90cda5b3a21306
Nov 03 19:48:26.626452 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.626429    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"]
Nov 03 19:48:26.626923 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.626892    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:48:26.626974 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.626961    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:48:26.627014 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.626994    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:48:26.862342 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.862306    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f5b246-ac88-4b7d-a616-093d1adf36c5" path="/var/lib/kubelet/pods/52f5b246-ac88-4b7d-a616-093d1adf36c5/volumes"
Nov 03 19:48:26.862865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.862846    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba12cce-ee41-46d3-a181-b01838b310cd" path="/var/lib/kubelet/pods/bba12cce-ee41-46d3-a181-b01838b310cd/volumes"
Nov 03 19:48:26.863358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:26.863338    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d1aab5-730f-4393-a956-d721f1a7fcc0" path="/var/lib/kubelet/pods/f7d1aab5-730f-4393-a956-d721f1a7fcc0/volumes"
Nov 03 19:48:27.232637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:27.230781    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:27.348659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:27.348617    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" event={"ID":"b63cceab-6dee-4689-8767-9a472d99b541","Type":"ContainerStarted","Data":"1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577"}
Nov 03 19:48:27.348848 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:27.348672    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" event={"ID":"b63cceab-6dee-4689-8767-9a472d99b541","Type":"ContainerStarted","Data":"b3c5cc652b35324bc46c14285dfe8888567de74ac6e556eb0c90cda5b3a21306"}
Nov 03 19:48:27.372790 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:27.372729    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" podStartSLOduration=2.372714956 podStartE2EDuration="2.372714956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:27.367671166 +0000 UTC m=+1020.637794460" watchObservedRunningTime="2026-11-03 19:48:27.372714956 +0000 UTC m=+1020.642838249"
Nov 03 19:48:28.060199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:28.054893    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:28.088198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:28.075671    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:28.115245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:28.114568    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:28.357913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:28.357827    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:28.358898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:28.358876    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:48:28.397380 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:28.397337    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:28.940416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:28.930668    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:29.604831 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:29.604795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:29.782221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:29.774784    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:30.377446 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:30.377352    4531 generic.go:356] "Generic (PLEG): container finished" podID="24a7070e-597a-4d4f-9f62-ea93932fffd3" containerID="9facc79e9d4af4a75b881d5f163186f5f41507504ca4b69009e4b8d6d78a49f3" exitCode=0
Nov 03 19:48:30.377446 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:30.377433    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7" event={"ID":"24a7070e-597a-4d4f-9f62-ea93932fffd3","Type":"ContainerDied","Data":"9facc79e9d4af4a75b881d5f163186f5f41507504ca4b69009e4b8d6d78a49f3"}
Nov 03 19:48:30.670336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:30.670298    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:30.885764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:30.885657    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:30.917671 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:30.917641    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"]
Nov 03 19:48:30.922650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:30.922587    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 19:48:30.925600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:30.925574    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-6eb768\""
Nov 03 19:48:30.925923 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:30.925905    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-6eb768-dockercfg-vx2jf\""
Nov 03 19:48:30.931401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:30.931377    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"]
Nov 03 19:48:30.934954 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:30.934933    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:48:31.092198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.088513    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-data\") pod \"24a7070e-597a-4d4f-9f62-ea93932fffd3\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") "
Nov 03 19:48:31.092198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.088576    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-credential-socket\") pod \"24a7070e-597a-4d4f-9f62-ea93932fffd3\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") "
Nov 03 19:48:31.092198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.088639    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-kube-api-access-lxt7b\" (UniqueName: \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-kube-api-access-lxt7b\") pod \"24a7070e-597a-4d4f-9f62-ea93932fffd3\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") "
Nov 03 19:48:31.121195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.117778    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-credential-socket" pod "24a7070e-597a-4d4f-9f62-ea93932fffd3" (UID: "24a7070e-597a-4d4f-9f62-ea93932fffd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:48:31.121195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.119295    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-socket\") pod \"24a7070e-597a-4d4f-9f62-ea93932fffd3\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") "
Nov 03 19:48:31.121195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.119491    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a7070e-597a-4d4f-9f62-ea93932fffd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a7070e-597a-4d4f-9f62-ea93932fffd3-istiod-ca-cert\") pod \"24a7070e-597a-4d4f-9f62-ea93932fffd3\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") "
Nov 03 19:48:31.121195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.119556    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-token\" (UniqueName: \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-token\") pod \"24a7070e-597a-4d4f-9f62-ea93932fffd3\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") "
Nov 03 19:48:31.121195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.119593    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-podinfo\") pod \"24a7070e-597a-4d4f-9f62-ea93932fffd3\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") "
Nov 03 19:48:31.121195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.119711    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-certs\") pod \"24a7070e-597a-4d4f-9f62-ea93932fffd3\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") "
Nov 03 19:48:31.121195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.119762    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-envoy\") pod \"24a7070e-597a-4d4f-9f62-ea93932fffd3\" (UID: \"24a7070e-597a-4d4f-9f62-ea93932fffd3\") "
Nov 03 19:48:31.121195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.120406    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987b5e63-c14d-473d-a5b6-bd5791338bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq\" (UID: \"987b5e63-c14d-473d-a5b6-bd5791338bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 19:48:31.121195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.120470    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pmkd\" (UniqueName: \"kubernetes.io/projected/987b5e63-c14d-473d-a5b6-bd5791338bce-kube-api-access-5pmkd\") pod \"payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq\" (UID: \"987b5e63-c14d-473d-a5b6-bd5791338bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 19:48:31.121195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.120757    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:31.154320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.146278    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-data" pod "24a7070e-597a-4d4f-9f62-ea93932fffd3" (UID: "24a7070e-597a-4d4f-9f62-ea93932fffd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:48:31.154320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.146806    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a7070e-597a-4d4f-9f62-ea93932fffd3-istiod-ca-cert" pod "24a7070e-597a-4d4f-9f62-ea93932fffd3" (UID: "24a7070e-597a-4d4f-9f62-ea93932fffd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:31.154320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.152017    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-socket" pod "24a7070e-597a-4d4f-9f62-ea93932fffd3" (UID: "24a7070e-597a-4d4f-9f62-ea93932fffd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:48:31.154320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.154269    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-certs" pod "24a7070e-597a-4d4f-9f62-ea93932fffd3" (UID: "24a7070e-597a-4d4f-9f62-ea93932fffd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:48:31.327829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.257954    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pmkd\" (UniqueName: \"kubernetes.io/projected/987b5e63-c14d-473d-a5b6-bd5791338bce-kube-api-access-5pmkd\") pod \"payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq\" (UID: \"987b5e63-c14d-473d-a5b6-bd5791338bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 19:48:31.358763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.354621    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987b5e63-c14d-473d-a5b6-bd5791338bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq\" (UID: \"987b5e63-c14d-473d-a5b6-bd5791338bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 19:48:31.358763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.354785    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:31.358763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.354801    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:31.358763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.354817    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:31.358763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.354833    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a7070e-597a-4d4f-9f62-ea93932fffd3-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:31.358763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.355733    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987b5e63-c14d-473d-a5b6-bd5791338bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq\" (UID: \"987b5e63-c14d-473d-a5b6-bd5791338bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 19:48:31.374826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.374343    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-envoy" pod "24a7070e-597a-4d4f-9f62-ea93932fffd3" (UID: "24a7070e-597a-4d4f-9f62-ea93932fffd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:48:31.374826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.374614    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-podinfo" pod "24a7070e-597a-4d4f-9f62-ea93932fffd3" (UID: "24a7070e-597a-4d4f-9f62-ea93932fffd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:48:31.374826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.374660    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-kube-api-access-lxt7b" pod "24a7070e-597a-4d4f-9f62-ea93932fffd3" (UID: "24a7070e-597a-4d4f-9f62-ea93932fffd3"). InnerVolumeSpecName "kube-api-access-lxt7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:31.399222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.399151    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"]
Nov 03 19:48:31.432797 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.432758    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:31.434452 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.434421    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-token" pod "24a7070e-597a-4d4f-9f62-ea93932fffd3" (UID: "24a7070e-597a-4d4f-9f62-ea93932fffd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:31.469307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.458765    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:31.469307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.458806    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:31.469307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.458823    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a7070e-597a-4d4f-9f62-ea93932fffd3-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:31.469307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.458838    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxt7b\" (UniqueName: \"kubernetes.io/projected/24a7070e-597a-4d4f-9f62-ea93932fffd3-kube-api-access-lxt7b\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:31.469307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.463367    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7" event={"ID":"24a7070e-597a-4d4f-9f62-ea93932fffd3","Type":"ContainerDied","Data":"ba5e080cf211040645bb554e83372f4eab88cc000dd8422a1a96324a27de8c35"}
Nov 03 19:48:31.469307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.463478    4531 scope.go:122] "RemoveContainer" containerID="9facc79e9d4af4a75b881d5f163186f5f41507504ca4b69009e4b8d6d78a49f3"
Nov 03 19:48:31.469307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.463699    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"
Nov 03 19:48:31.472792 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.472762    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c684deb-fwnvz"]
Nov 03 19:48:31.472884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.472819    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pmkd\" (UniqueName: \"kubernetes.io/projected/987b5e63-c14d-473d-a5b6-bd5791338bce-kube-api-access-5pmkd\") pod \"payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq\" (UID: \"987b5e63-c14d-473d-a5b6-bd5791338bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 19:48:31.502470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.502441    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"]
Nov 03 19:48:31.508357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.508334    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c96ee058-gw-openshift-default-d47865b85-m2kp7"]
Nov 03 19:48:31.555184 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.555137    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 19:48:31.695192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.691689    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:31.849561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:31.849536    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"]
Nov 03 19:48:32.341745 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.341709    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:32.475606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.475572    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq" event={"ID":"987b5e63-c14d-473d-a5b6-bd5791338bce","Type":"ContainerStarted","Data":"519f38ac13e81ec067a9459c56a8df544ccb999024e558541a138ceea821f56c"}
Nov 03 19:48:32.475766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.475623    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq" event={"ID":"987b5e63-c14d-473d-a5b6-bd5791338bce","Type":"ContainerStarted","Data":"de3de870f6ed8ae42dd996bd6a4fe9c5c2937b0925ae94bc9ac69a4242594040"}
Nov 03 19:48:32.475766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.475703    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 19:48:32.494873 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.494818    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq" podStartSLOduration=2.494803669 podStartE2EDuration="2.494803669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:32.488653907 +0000 UTC m=+1025.758777201" watchObservedRunningTime="2026-11-03 19:48:32.494803669 +0000 UTC m=+1025.764926964"
Nov 03 19:48:32.572387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.572344    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"]
Nov 03 19:48:32.584281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.584248    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"]
Nov 03 19:48:32.694185 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.694139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:32.789241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.789203    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"]
Nov 03 19:48:32.789683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.789562    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t" podUID="3f42caee-4e6c-4d70-b9b3-18490efc7ea1" containerName="maas-api" containerID="cri-o://ec222eafe1649741cf21708519e48d7dc552a9cb90e27c5800959e38424cbc8d" gracePeriod=30
Nov 03 19:48:32.822537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.822508    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:32.860810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.860779    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a7070e-597a-4d4f-9f62-ea93932fffd3" path="/var/lib/kubelet/pods/24a7070e-597a-4d4f-9f62-ea93932fffd3/volumes"
Nov 03 19:48:32.861462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.861442    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75ded06-8e99-4a28-be0b-8de4717a6999" path="/var/lib/kubelet/pods/c75ded06-8e99-4a28-be0b-8de4717a6999/volumes"
Nov 03 19:48:32.954142 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:32.953010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:33.309576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.301855    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:33.331716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.317427    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"]
Nov 03 19:48:33.331716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.323250    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af73ad23-44vwp"]
Nov 03 19:48:33.503127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.503082    4531 generic.go:356] "Generic (PLEG): container finished" podID="3f42caee-4e6c-4d70-b9b3-18490efc7ea1" containerID="ec222eafe1649741cf21708519e48d7dc552a9cb90e27c5800959e38424cbc8d" exitCode=0
Nov 03 19:48:33.504427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.504402    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t" event={"ID":"3f42caee-4e6c-4d70-b9b3-18490efc7ea1","Type":"ContainerDied","Data":"ec222eafe1649741cf21708519e48d7dc552a9cb90e27c5800959e38424cbc8d"}
Nov 03 19:48:33.558525 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.558434    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:48:33.591989 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.591908    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-metrics-certs\") pod \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") "
Nov 03 19:48:33.591989 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.591962    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-openshift-service-ca\") pod \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") "
Nov 03 19:48:33.592251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.592233    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-maas-api-tls\") pod \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") "
Nov 03 19:48:33.592348 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.592332    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-kube-api-access-zgtg2\" (UniqueName: \"kubernetes.io/projected/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-kube-api-access-zgtg2\") pod \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\" (UID: \"3f42caee-4e6c-4d70-b9b3-18490efc7ea1\") "
Nov 03 19:48:33.592434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.592410    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-openshift-service-ca" pod "3f42caee-4e6c-4d70-b9b3-18490efc7ea1" (UID: "3f42caee-4e6c-4d70-b9b3-18490efc7ea1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:33.593334 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.593310    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:33.594782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.594760    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-maas-api-tls" pod "3f42caee-4e6c-4d70-b9b3-18490efc7ea1" (UID: "3f42caee-4e6c-4d70-b9b3-18490efc7ea1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:33.594886 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.594813    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-kube-api-access-zgtg2" pod "3f42caee-4e6c-4d70-b9b3-18490efc7ea1" (UID: "3f42caee-4e6c-4d70-b9b3-18490efc7ea1"). InnerVolumeSpecName "kube-api-access-zgtg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:33.595310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.595289    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-metrics-certs" pod "3f42caee-4e6c-4d70-b9b3-18490efc7ea1" (UID: "3f42caee-4e6c-4d70-b9b3-18490efc7ea1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:33.702084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.701998    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:33.702084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.702044    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgtg2\" (UniqueName: \"kubernetes.io/projected/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-kube-api-access-zgtg2\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:33.702084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.702078    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f42caee-4e6c-4d70-b9b3-18490efc7ea1-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:33.977198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:33.969311    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:34.447287 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:34.447238    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:34.513704 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:34.513673    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"
Nov 03 19:48:34.513890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:34.513681    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t" event={"ID":"3f42caee-4e6c-4d70-b9b3-18490efc7ea1","Type":"ContainerDied","Data":"1f37f4c794571934277cd762a2d3d392908ebc581fcc5c48bc3158c43ffdf2c3"}
Nov 03 19:48:34.513890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:34.513813    4531 scope.go:122] "RemoveContainer" containerID="ec222eafe1649741cf21708519e48d7dc552a9cb90e27c5800959e38424cbc8d"
Nov 03 19:48:34.538260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:34.538235    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"]
Nov 03 19:48:34.543886 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:34.543863    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t"]
Nov 03 19:48:34.683047 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:34.683014    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:34.861101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:34.861023    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f42caee-4e6c-4d70-b9b3-18490efc7ea1" path="/var/lib/kubelet/pods/3f42caee-4e6c-4d70-b9b3-18490efc7ea1/volumes"
Nov 03 19:48:34.861601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:34.861584    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad958fee-4506-4ec9-b3a2-1f9913595aad" path="/var/lib/kubelet/pods/ad958fee-4506-4ec9-b3a2-1f9913595aad/volumes"
Nov 03 19:48:35.254576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.254543    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"]
Nov 03 19:48:35.255834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.255658    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3f42caee-4e6c-4d70-b9b3-18490efc7ea1" containerName="maas-api"
Nov 03 19:48:35.255834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.255683    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f42caee-4e6c-4d70-b9b3-18490efc7ea1" containerName="maas-api"
Nov 03 19:48:35.255834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.255771    4531 cpu_manager.go:414] "containerMap: removing container" podUID="24a7070e-597a-4d4f-9f62-ea93932fffd3" containerName="istio-proxy"
Nov 03 19:48:35.255834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.255778    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a7070e-597a-4d4f-9f62-ea93932fffd3" containerName="istio-proxy"
Nov 03 19:48:35.256090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.255995    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a7070e-597a-4d4f-9f62-ea93932fffd3" containerName="istio-proxy"
Nov 03 19:48:35.256090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.256016    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f42caee-4e6c-4d70-b9b3-18490efc7ea1" containerName="maas-api"
Nov 03 19:48:35.264759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.264732    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.267137 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.267111    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"]
Nov 03 19:48:35.272492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.272468    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-6eb768-serving-cert\""
Nov 03 19:48:35.335345 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.335308    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.335545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.335366    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9377699d-5e72-4b01-966b-b9a4686c6859-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.335710 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.335684    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpkv\" (UniqueName: \"kubernetes.io/projected/9377699d-5e72-4b01-966b-b9a4686c6859-kube-api-access-8lpkv\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.335776 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.335760    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.347035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.347010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:35.437126 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.437092    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.437312 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.437143    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9377699d-5e72-4b01-966b-b9a4686c6859-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.437411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.437396    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpkv\" (UniqueName: \"kubernetes.io/projected/9377699d-5e72-4b01-966b-b9a4686c6859-kube-api-access-8lpkv\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.437463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.437438    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.437907 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.437880    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9377699d-5e72-4b01-966b-b9a4686c6859-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.439837 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.439813    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.439952 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.439887    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.450399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.447321    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpkv\" (UniqueName: \"kubernetes.io/projected/9377699d-5e72-4b01-966b-b9a4686c6859-kube-api-access-8lpkv\") pod \"maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.487218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.485485    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:35.589284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.588570    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:35.963250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:35.961000    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"]
Nov 03 19:48:36.057241 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:48:36.053453    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9377699d_5e72_4b01_966b_b9a4686c6859.slice/crio-59c1b5d3a4b49cea76b54b7526325c8a3bf81145d539586cbcc4e11bbb22530e WatchSource:0}: Error finding container 59c1b5d3a4b49cea76b54b7526325c8a3bf81145d539586cbcc4e11bbb22530e: Status 404 returned error can't find the container with id 59c1b5d3a4b49cea76b54b7526325c8a3bf81145d539586cbcc4e11bbb22530e
Nov 03 19:48:36.172167 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:36.172131    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b3a1058c-57c4946f58-pbn5t" podUID="3f42caee-4e6c-4d70-b9b3-18490efc7ea1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.140:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:36.222656 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:36.222581    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:36.644223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:36.641331    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8" event={"ID":"9377699d-5e72-4b01-966b-b9a4686c6859","Type":"ContainerStarted","Data":"59c1b5d3a4b49cea76b54b7526325c8a3bf81145d539586cbcc4e11bbb22530e"}
Nov 03 19:48:36.949209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:36.948633    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:37.572901 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:37.572849    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth" podUID="0b22a5cc-b303-4db6-b943-61473387c8b4" containerName="payload-processing" containerID="cri-o://940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25" gracePeriod=25
Nov 03 19:48:37.584741 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:37.584690    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn" podUID="c4a546ae-c169-464c-b16c-8667bd96879b" containerName="payload-pre-processing" containerID="cri-o://1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd" gracePeriod=25
Nov 03 19:48:37.719162 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:37.718813    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:37.725310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:37.722662    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8" event={"ID":"9377699d-5e72-4b01-966b-b9a4686c6859","Type":"ContainerStarted","Data":"7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083"}
Nov 03 19:48:37.725310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:37.724544    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:37.743647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:37.743601    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8" podStartSLOduration=2.305305495 podStartE2EDuration="2.74358599s" totalImagesPullingTime="438.280495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:37.73950086 +0000 UTC m=+1031.009624156" watchObservedRunningTime="2026-11-03 19:48:37.74358599 +0000 UTC m=+1031.013709283"
Nov 03 19:48:38.029538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.029450    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:48:38.049878 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.049004    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:48:38.136291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.136262    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a546ae-c169-464c-b16c-8667bd96879b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a546ae-c169-464c-b16c-8667bd96879b-plugins-config-volume\") pod \"c4a546ae-c169-464c-b16c-8667bd96879b\" (UID: \"c4a546ae-c169-464c-b16c-8667bd96879b\") "
Nov 03 19:48:38.136291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.136297    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a546ae-c169-464c-b16c-8667bd96879b-kube-api-access-sncdr\" (UniqueName: \"kubernetes.io/projected/c4a546ae-c169-464c-b16c-8667bd96879b-kube-api-access-sncdr\") pod \"c4a546ae-c169-464c-b16c-8667bd96879b\" (UID: \"c4a546ae-c169-464c-b16c-8667bd96879b\") "
Nov 03 19:48:38.136559 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.136340    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b22a5cc-b303-4db6-b943-61473387c8b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b22a5cc-b303-4db6-b943-61473387c8b4-plugins-config-volume\") pod \"0b22a5cc-b303-4db6-b943-61473387c8b4\" (UID: \"0b22a5cc-b303-4db6-b943-61473387c8b4\") "
Nov 03 19:48:38.136559 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.136462    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b22a5cc-b303-4db6-b943-61473387c8b4-kube-api-access-77d22\" (UniqueName: \"kubernetes.io/projected/0b22a5cc-b303-4db6-b943-61473387c8b4-kube-api-access-77d22\") pod \"0b22a5cc-b303-4db6-b943-61473387c8b4\" (UID: \"0b22a5cc-b303-4db6-b943-61473387c8b4\") "
Nov 03 19:48:38.136725 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.136701    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a546ae-c169-464c-b16c-8667bd96879b-plugins-config-volume" pod "c4a546ae-c169-464c-b16c-8667bd96879b" (UID: "c4a546ae-c169-464c-b16c-8667bd96879b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:38.136777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.136711    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b22a5cc-b303-4db6-b943-61473387c8b4-plugins-config-volume" pod "0b22a5cc-b303-4db6-b943-61473387c8b4" (UID: "0b22a5cc-b303-4db6-b943-61473387c8b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:38.137141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.137117    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a546ae-c169-464c-b16c-8667bd96879b-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:38.137141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.137140    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b22a5cc-b303-4db6-b943-61473387c8b4-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:38.138585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.138558    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b22a5cc-b303-4db6-b943-61473387c8b4-kube-api-access-77d22" pod "0b22a5cc-b303-4db6-b943-61473387c8b4" (UID: "0b22a5cc-b303-4db6-b943-61473387c8b4"). InnerVolumeSpecName "kube-api-access-77d22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:38.138707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.138688    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a546ae-c169-464c-b16c-8667bd96879b-kube-api-access-sncdr" pod "c4a546ae-c169-464c-b16c-8667bd96879b" (UID: "c4a546ae-c169-464c-b16c-8667bd96879b"). InnerVolumeSpecName "kube-api-access-sncdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:38.238535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.238458    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77d22\" (UniqueName: \"kubernetes.io/projected/0b22a5cc-b303-4db6-b943-61473387c8b4-kube-api-access-77d22\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:38.238535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.238487    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sncdr\" (UniqueName: \"kubernetes.io/projected/c4a546ae-c169-464c-b16c-8667bd96879b-kube-api-access-sncdr\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:38.342200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.327055    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:38.733695 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.733661    4531 generic.go:356] "Generic (PLEG): container finished" podID="0b22a5cc-b303-4db6-b943-61473387c8b4" containerID="940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25" exitCode=0
Nov 03 19:48:38.734249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.733739    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"
Nov 03 19:48:38.734249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.733747    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth" event={"ID":"0b22a5cc-b303-4db6-b943-61473387c8b4","Type":"ContainerDied","Data":"940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25"}
Nov 03 19:48:38.734249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.733830    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth" event={"ID":"0b22a5cc-b303-4db6-b943-61473387c8b4","Type":"ContainerDied","Data":"8ab426db2157f3e7a6ff37a5e9ad7aea9e1c09867caebfa06e15cc06db04cc31"}
Nov 03 19:48:38.734249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.733852    4531 scope.go:122] "RemoveContainer" containerID="940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25"
Nov 03 19:48:38.735547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.735522    4531 generic.go:356] "Generic (PLEG): container finished" podID="c4a546ae-c169-464c-b16c-8667bd96879b" containerID="1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd" exitCode=0
Nov 03 19:48:38.735650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.735584    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"
Nov 03 19:48:38.735650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.735602    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn" event={"ID":"c4a546ae-c169-464c-b16c-8667bd96879b","Type":"ContainerDied","Data":"1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd"}
Nov 03 19:48:38.735770 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.735647    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn" event={"ID":"c4a546ae-c169-464c-b16c-8667bd96879b","Type":"ContainerDied","Data":"ea01029c8efa09e953fbae85c2d4173c93cc3d0877b2358913456a2062bdf0a2"}
Nov 03 19:48:38.749924 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.749898    4531 scope.go:122] "RemoveContainer" containerID="940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25"
Nov 03 19:48:38.750569 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:48:38.750534    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25\": container with ID starting with 940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25 not found: ID does not exist" containerID="940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25"
Nov 03 19:48:38.750672 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.750576    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25"} err="failed to get container status \"940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25\": rpc error: code = NotFound desc = could not find container \"940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25\": container with ID starting with 940a4ac1ca795910a380dfcf6bd2a0bf8fddb09967ef807340f3e5e7e35e6a25 not found: ID does not exist"
Nov 03 19:48:38.750672 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.750599    4531 scope.go:122] "RemoveContainer" containerID="1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd"
Nov 03 19:48:38.764096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.764071    4531 scope.go:122] "RemoveContainer" containerID="1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd"
Nov 03 19:48:38.764497 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:48:38.764462    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd\": container with ID starting with 1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd not found: ID does not exist" containerID="1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd"
Nov 03 19:48:38.764614 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.764505    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd"} err="failed to get container status \"1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd\": rpc error: code = NotFound desc = could not find container \"1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd\": container with ID starting with 1eb1cf74994b92f3902a372d0dd39310b6202fbad1697ae75533a950c6fa33bd not found: ID does not exist"
Nov 03 19:48:38.767513 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.767493    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"]
Nov 03 19:48:38.774113 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.774086    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b3a1058c-654dd64c97-qqmth"]
Nov 03 19:48:38.779899 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.779882    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"]
Nov 03 19:48:38.785267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.785246    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b3a1058c-7497754ff5-jpdmn"]
Nov 03 19:48:38.860945 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.860873    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b22a5cc-b303-4db6-b943-61473387c8b4" path="/var/lib/kubelet/pods/0b22a5cc-b303-4db6-b943-61473387c8b4/volumes"
Nov 03 19:48:38.861577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:38.861559    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a546ae-c169-464c-b16c-8667bd96879b" path="/var/lib/kubelet/pods/c4a546ae-c169-464c-b16c-8667bd96879b/volumes"
Nov 03 19:48:39.013558 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:39.013510    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:39.635735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:39.635649    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:39.760886 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:39.760855    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:39.969091 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:39.969009    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:40.225187 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:40.225081    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"]
Nov 03 19:48:40.225794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:40.225408    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5" podUID="a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" containerName="istio-proxy" containerID="cri-o://193c6959f1c790fe85f81a1ff398cfddb086bc5ee4035a4f11fd4497d441fbac" gracePeriod=30
Nov 03 19:48:40.303583 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:40.303549    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:40.855184 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:40.850854    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:41.054096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.054045    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:41.404913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.404865    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"]
Nov 03 19:48:41.405795 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.405772    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0b22a5cc-b303-4db6-b943-61473387c8b4" containerName="payload-processing"
Nov 03 19:48:41.405888 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.405797    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b22a5cc-b303-4db6-b943-61473387c8b4" containerName="payload-processing"
Nov 03 19:48:41.405888 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.405841    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c4a546ae-c169-464c-b16c-8667bd96879b" containerName="payload-pre-processing"
Nov 03 19:48:41.405888 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.405847    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a546ae-c169-464c-b16c-8667bd96879b" containerName="payload-pre-processing"
Nov 03 19:48:41.406045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.405990    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a546ae-c169-464c-b16c-8667bd96879b" containerName="payload-pre-processing"
Nov 03 19:48:41.406045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.406021    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b22a5cc-b303-4db6-b943-61473387c8b4" containerName="payload-processing"
Nov 03 19:48:41.418951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.418903    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.422996 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.422950    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-fcddeb-openshift-default-dockercfg-cq459\""
Nov 03 19:48:41.423581 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.423298    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"]
Nov 03 19:48:41.448750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.448716    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.448871 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.448794    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.448957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.448915    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpd4\" (UniqueName: \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-kube-api-access-hwpd4\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.449101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.449085    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.449187 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.449127    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.449254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.449214    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.449312 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.449253    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385aa4c7-24cc-4a11-a69b-e572797c115f-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.449415 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.449396    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.449473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.449441    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.456461 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.456416    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:41.580196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.577295    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.580196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.577352    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.580196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.577405    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.580196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.577437    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385aa4c7-24cc-4a11-a69b-e572797c115f-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.580196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.577518    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.580196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.577556    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.580196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.577601    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.580196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.577648    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.580196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.577745    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpd4\" (UniqueName: \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-kube-api-access-hwpd4\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.589804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.589769    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.661246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.660383    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpd4\" (UniqueName: \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-kube-api-access-hwpd4\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.679196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.662062    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:41.683337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.679580    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.700197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.693761    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.701594 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.701564    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.737195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.728556    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.737195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.728901    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.737195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.729475    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385aa4c7-24cc-4a11-a69b-e572797c115f-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:41.852121 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.852081    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:41.858197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:41.854166    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:42.055545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:42.053494    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:42.439707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:42.439670    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:42.564526 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:48:42.564478    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385aa4c7_24cc_4a11_a69b_e572797c115f.slice/crio-ef2d829221ffb02ba2b52c0731e7dd323898bc5c1999da801777fa73c5a24d79 WatchSource:0}: Error finding container ef2d829221ffb02ba2b52c0731e7dd323898bc5c1999da801777fa73c5a24d79: Status 404 returned error can't find the container with id ef2d829221ffb02ba2b52c0731e7dd323898bc5c1999da801777fa73c5a24d79
Nov 03 19:48:42.567379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:42.567343    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:48:42.567485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:42.567412    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:48:42.567485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:42.567442    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:48:42.568489 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:42.568466    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"]
Nov 03 19:48:42.873229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:42.873115    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" event={"ID":"385aa4c7-24cc-4a11-a69b-e572797c115f","Type":"ContainerStarted","Data":"a9750e8a6fd96da631ad0f22a2e3f730b213834905cd3fe34b63fab28b7e2e91"}
Nov 03 19:48:42.873624 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:42.873221    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" event={"ID":"385aa4c7-24cc-4a11-a69b-e572797c115f","Type":"ContainerStarted","Data":"ef2d829221ffb02ba2b52c0731e7dd323898bc5c1999da801777fa73c5a24d79"}
Nov 03 19:48:42.979211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:42.974486    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" podStartSLOduration=1.9744619810000001 podStartE2EDuration="1.974461981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:42.904987456 +0000 UTC m=+1036.175110809" watchObservedRunningTime="2026-11-03 19:48:42.974461981 +0000 UTC m=+1036.244585278"
Nov 03 19:48:43.059197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:43.058404    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:43.082231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:43.068294    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 19:48:43.082231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:43.068368    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 19:48:43.373488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:43.370020    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:43.510293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:43.509963    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 19:48:44.028689 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:44.028649    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"]
Nov 03 19:48:44.034944 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:44.034921    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90922dc8-lwqk8"]
Nov 03 19:48:44.061266 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:44.061241    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:44.293767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:44.293677    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:44.763217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:44.763139    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 19:48:44.872062 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:44.871293    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ba0b5a-0ffe-41a7-8925-a886f068a1b3" path="/var/lib/kubelet/pods/e4ba0b5a-0ffe-41a7-8925-a886f068a1b3/volumes"
Nov 03 19:48:45.016298 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:45.003494    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:45.108195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:45.103094    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:48:45.491438 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:45.487667    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:45.762808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:45.762774    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:46.099582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:46.099492    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:46.120259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:46.118207    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:46.390707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:46.277604    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:46.466198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:46.405193    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:48:46.714258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:46.688563    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:48:46.714258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:46.689968    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:48:46.719311 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:46.718058    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:48:46.814586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:46.792349    4531 generic.go:356] "Generic (PLEG): container finished" podID="a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" containerID="193c6959f1c790fe85f81a1ff398cfddb086bc5ee4035a4f11fd4497d441fbac" exitCode=0
Nov 03 19:48:46.814586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:46.793224    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5" event={"ID":"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a","Type":"ContainerDied","Data":"193c6959f1c790fe85f81a1ff398cfddb086bc5ee4035a4f11fd4497d441fbac"}
Nov 03 19:48:46.895198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:46.895040    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:48:46.940281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:46.924871    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:47.137232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.136876    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:47.308560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.247727    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:47.308560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.256681    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:48:47.308560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.282125    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:47.336974 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.336938    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:48:47.373229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.372860    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-envoy\") pod \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") "
Nov 03 19:48:47.373229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.372931    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-credential-socket\") pod \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") "
Nov 03 19:48:47.373229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.373026    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-podinfo\") pod \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") "
Nov 03 19:48:47.373229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.373063    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istiod-ca-cert\") pod \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") "
Nov 03 19:48:47.373229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.373108    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-data\") pod \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") "
Nov 03 19:48:47.406628 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.397484    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-data" pod "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" (UID: "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:48:47.433196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.409602    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istiod-ca-cert" pod "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" (UID: "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:47.433196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.411243    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-kube-api-access-94wc6\" (UniqueName: \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-kube-api-access-94wc6\") pod \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") "
Nov 03 19:48:47.433196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.411344    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-token\" (UniqueName: \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-token\") pod \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") "
Nov 03 19:48:47.433196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.411393    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-socket\") pod \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") "
Nov 03 19:48:47.433196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.411574    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-certs\") pod \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\" (UID: \"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a\") "
Nov 03 19:48:47.433196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.412693    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:47.433196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.412720    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:47.433196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.425788    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-certs" pod "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" (UID: "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:48:47.472193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.444642    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-credential-socket" pod "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" (UID: "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:48:47.493848 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.493768    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-socket" pod "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" (UID: "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:48:47.504527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.499951    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:48:47.519943 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.518990    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:47.519943 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.519031    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:47.519943 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.519049    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:47.656247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.541266    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"]
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.679507    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7deb052d-6k674"]
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.679592    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.679639    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.679671    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.679688    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"]
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.680905    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" containerName="istio-proxy"
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.680925    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" containerName="istio-proxy"
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.681162    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" containerName="istio-proxy"
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.674101    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.735315    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.736000    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-kube-api-access-94wc6" pod "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" (UID: "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a"). InnerVolumeSpecName "kube-api-access-94wc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.736113    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-podinfo" pod "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" (UID: "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.741645    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.742168    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.742260    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.742604    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 19:48:47.751536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.750551    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"]
Nov 03 19:48:47.760401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.758514    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-fcddeb-dockercfg-57f5q\""
Nov 03 19:48:47.760401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.759414    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-fcddeb\""
Nov 03 19:48:47.760401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.759957    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:48:47.826638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.824594    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a4e015-a44e-4628-9f0c-101b362975cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd\" (UID: \"00a4e015-a44e-4628-9f0c-101b362975cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 19:48:47.826638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.825123    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6sk\" (UniqueName: \"kubernetes.io/projected/00a4e015-a44e-4628-9f0c-101b362975cb-kube-api-access-6m6sk\") pod \"payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd\" (UID: \"00a4e015-a44e-4628-9f0c-101b362975cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 19:48:47.826638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.826046    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:47.826638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.826342    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94wc6\" (UniqueName: \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-kube-api-access-94wc6\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:47.838222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.829348    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"]
Nov 03 19:48:47.850324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.847416    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"]
Nov 03 19:48:47.850324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.847546    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 19:48:47.869330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.869305    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:48:47.883403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.883367    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-token" pod "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" (UID: "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:47.887518 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.887485    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-envoy" pod "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" (UID: "a7d3e888-4314-4c8a-b4a4-cad39c13ee1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:48:47.900344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.899633    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:48:47.902992 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.900685    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:48:47.902992 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.902097    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:48:47.904499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.904466    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"
Nov 03 19:48:47.904777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.904638    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5" event={"ID":"a7d3e888-4314-4c8a-b4a4-cad39c13ee1a","Type":"ContainerDied","Data":"1b06d26d1d7b50e0b726f28808b125523e9800117d5e488d7f8742b02e06f5b5"}
Nov 03 19:48:47.904965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.904953    4531 scope.go:122] "RemoveContainer" containerID="193c6959f1c790fe85f81a1ff398cfddb086bc5ee4035a4f11fd4497d441fbac"
Nov 03 19:48:47.935499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.935467    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a4e015-a44e-4628-9f0c-101b362975cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd\" (UID: \"00a4e015-a44e-4628-9f0c-101b362975cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 19:48:47.936021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.935990    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6sk\" (UniqueName: \"kubernetes.io/projected/00a4e015-a44e-4628-9f0c-101b362975cb-kube-api-access-6m6sk\") pod \"payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd\" (UID: \"00a4e015-a44e-4628-9f0c-101b362975cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 19:48:47.936299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.936282    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:47.936434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.936420    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:47.939579 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.939542    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a4e015-a44e-4628-9f0c-101b362975cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd\" (UID: \"00a4e015-a44e-4628-9f0c-101b362975cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 19:48:47.957343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.957311    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6sk\" (UniqueName: \"kubernetes.io/projected/00a4e015-a44e-4628-9f0c-101b362975cb-kube-api-access-6m6sk\") pod \"payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd\" (UID: \"00a4e015-a44e-4628-9f0c-101b362975cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 19:48:47.986493 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.986460    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"]
Nov 03 19:48:47.993866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:47.993840    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b3a1058c-openshift-default-66948f8c99-ng8b5"]
Nov 03 19:48:48.039122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.039022    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b84f64d-2622-4d11-aed6-ad27e8b50125-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8\" (UID: \"9b84f64d-2622-4d11-aed6-ad27e8b50125\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 19:48:48.039777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.039429    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4z48\" (UniqueName: \"kubernetes.io/projected/9b84f64d-2622-4d11-aed6-ad27e8b50125-kube-api-access-s4z48\") pod \"payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8\" (UID: \"9b84f64d-2622-4d11-aed6-ad27e8b50125\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 19:48:48.140570 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.140535    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b84f64d-2622-4d11-aed6-ad27e8b50125-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8\" (UID: \"9b84f64d-2622-4d11-aed6-ad27e8b50125\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 19:48:48.140981 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.140736    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4z48\" (UniqueName: \"kubernetes.io/projected/9b84f64d-2622-4d11-aed6-ad27e8b50125-kube-api-access-s4z48\") pod \"payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8\" (UID: \"9b84f64d-2622-4d11-aed6-ad27e8b50125\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 19:48:48.141493 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.141468    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b84f64d-2622-4d11-aed6-ad27e8b50125-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8\" (UID: \"9b84f64d-2622-4d11-aed6-ad27e8b50125\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 19:48:48.148410 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.148390    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4z48\" (UniqueName: \"kubernetes.io/projected/9b84f64d-2622-4d11-aed6-ad27e8b50125-kube-api-access-s4z48\") pod \"payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8\" (UID: \"9b84f64d-2622-4d11-aed6-ad27e8b50125\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 19:48:48.182165 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.182125    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 19:48:48.244114 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.244081    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 19:48:48.341029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.340636    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:48.483122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.483091    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"]
Nov 03 19:48:48.487954 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:48:48.487923    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a4e015_a44e_4628_9f0c_101b362975cb.slice/crio-d67eca302b86249801bdd48f7a4d9d12f9d4b5804846b8b8a3f9c2acedbd3acf WatchSource:0}: Error finding container d67eca302b86249801bdd48f7a4d9d12f9d4b5804846b8b8a3f9c2acedbd3acf: Status 404 returned error can't find the container with id d67eca302b86249801bdd48f7a4d9d12f9d4b5804846b8b8a3f9c2acedbd3acf
Nov 03 19:48:48.542890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.542866    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"]
Nov 03 19:48:48.545676 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:48:48.545653    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b84f64d_2622_4d11_aed6_ad27e8b50125.slice/crio-9dbc0622527875a3c653fc2686db31fa197a7e91344afc4a0d37672efafef556 WatchSource:0}: Error finding container 9dbc0622527875a3c653fc2686db31fa197a7e91344afc4a0d37672efafef556: Status 404 returned error can't find the container with id 9dbc0622527875a3c653fc2686db31fa197a7e91344afc4a0d37672efafef556
Nov 03 19:48:48.860857 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.860773    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40baf39d-7d38-4997-9e5e-9f4f35005183" path="/var/lib/kubelet/pods/40baf39d-7d38-4997-9e5e-9f4f35005183/volumes"
Nov 03 19:48:48.861347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.861332    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d3e888-4314-4c8a-b4a4-cad39c13ee1a" path="/var/lib/kubelet/pods/a7d3e888-4314-4c8a-b4a4-cad39c13ee1a/volumes"
Nov 03 19:48:48.921884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.921847    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8" event={"ID":"9b84f64d-2622-4d11-aed6-ad27e8b50125","Type":"ContainerStarted","Data":"3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2"}
Nov 03 19:48:48.922056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.921912    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8" event={"ID":"9b84f64d-2622-4d11-aed6-ad27e8b50125","Type":"ContainerStarted","Data":"9dbc0622527875a3c653fc2686db31fa197a7e91344afc4a0d37672efafef556"}
Nov 03 19:48:48.922056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.921943    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 19:48:48.924707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.924682    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd" event={"ID":"00a4e015-a44e-4628-9f0c-101b362975cb","Type":"ContainerStarted","Data":"445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39"}
Nov 03 19:48:48.924864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.924730    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd" event={"ID":"00a4e015-a44e-4628-9f0c-101b362975cb","Type":"ContainerStarted","Data":"d67eca302b86249801bdd48f7a4d9d12f9d4b5804846b8b8a3f9c2acedbd3acf"}
Nov 03 19:48:48.925119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.925096    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 19:48:48.936944 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.936893    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8" podStartSLOduration=1.936876356 podStartE2EDuration="1.936876356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:48.935599706 +0000 UTC m=+1042.205723001" watchObservedRunningTime="2026-11-03 19:48:48.936876356 +0000 UTC m=+1042.206999651"
Nov 03 19:48:48.953953 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:48.953889    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd" podStartSLOduration=1.953875142 podStartE2EDuration="1.953875142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:48.949772062 +0000 UTC m=+1042.219895356" watchObservedRunningTime="2026-11-03 19:48:48.953875142 +0000 UTC m=+1042.223998436"
Nov 03 19:48:49.140934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:49.140791    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:49.821753 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:49.821629    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:50.434841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:50.434799    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:50.586744 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:48:50.586704    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:48:51.025045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:51.025012    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"]
Nov 03 19:48:51.031107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:51.031077    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e189962-pggfx"]
Nov 03 19:48:51.147813 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:51.147761    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:51.954206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:51.938680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:52.064401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.064367    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"]
Nov 03 19:48:52.086734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.086697    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2592cfd8-sj9pt"]
Nov 03 19:48:52.540911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.540879    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"]
Nov 03 19:48:52.549873 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.549845    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.555890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.555856    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"]
Nov 03 19:48:52.561307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.561279    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-fcddeb-serving-cert\""
Nov 03 19:48:52.631160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.631127    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:52.634106 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.634079    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.634336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.634308    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqks\" (UniqueName: \"kubernetes.io/projected/211d2b1f-7806-4f29-8c89-35f05f3cd673-kube-api-access-6gqks\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.634482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.634461    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.634526 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.634506    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211d2b1f-7806-4f29-8c89-35f05f3cd673-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.742098 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.742057    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.742510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.742481    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211d2b1f-7806-4f29-8c89-35f05f3cd673-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.743081 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.743059    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.748485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.748459    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqks\" (UniqueName: \"kubernetes.io/projected/211d2b1f-7806-4f29-8c89-35f05f3cd673-kube-api-access-6gqks\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.749230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.744397    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211d2b1f-7806-4f29-8c89-35f05f3cd673-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.763205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.758874    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:52.776195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.767965    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqks\" (UniqueName: \"kubernetes.io/projected/211d2b1f-7806-4f29-8c89-35f05f3cd673-kube-api-access-6gqks\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.776195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.769996    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.776195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.775422    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.861625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.861543    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197e6009-fb6b-4432-a8d6-f0cdfeff4987" path="/var/lib/kubelet/pods/197e6009-fb6b-4432-a8d6-f0cdfeff4987/volumes"
Nov 03 19:48:52.862188 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.862153    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716a9aa4-ec8a-47cf-bd77-7e616298e2fb" path="/var/lib/kubelet/pods/716a9aa4-ec8a-47cf-bd77-7e616298e2fb/volumes"
Nov 03 19:48:52.881579 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.881554    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:52.891271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:52.891246    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:53.253204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:53.250871    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"]
Nov 03 19:48:53.423548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:53.423510    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:53.677665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:53.677627    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:53.974763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:53.963458    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:54.106116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:54.105840    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr" event={"ID":"211d2b1f-7806-4f29-8c89-35f05f3cd673","Type":"ContainerStarted","Data":"9aa09a1b27dce6f9fdfdd7e81d0eb2b138f6ff381c525f1a9702129db9d0e28c"}
Nov 03 19:48:54.455295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:54.455252    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:54.837256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:54.828837    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:55.168658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:55.167604    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr" event={"ID":"211d2b1f-7806-4f29-8c89-35f05f3cd673","Type":"ContainerStarted","Data":"d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13"}
Nov 03 19:48:55.168658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:55.168259    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:48:55.190101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:55.190045    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr" podStartSLOduration=2.681837835 podStartE2EDuration="3.190019163s" totalImagesPullingTime="508.181328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:55.184098572 +0000 UTC m=+1048.454221867" watchObservedRunningTime="2026-11-03 19:48:55.190019163 +0000 UTC m=+1048.460142457"
Nov 03 19:48:55.233673 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:55.233640    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:55.393278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:55.391188    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:55.663382 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:55.663345    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:55.807621 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:55.807583    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:56.353206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:56.349747    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:57.025270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:57.025228    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"]
Nov 03 19:48:57.037867 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:57.037825    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e8caeac-sx5vk"]
Nov 03 19:48:57.072890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:57.072839    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:57.741199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:57.740967    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:58.666416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:58.658741    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:58.861308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:58.861273    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bf8c2c-a57e-47a9-b596-30eda5588e61" path="/var/lib/kubelet/pods/64bf8c2c-a57e-47a9-b596-30eda5588e61/volumes"
Nov 03 19:48:59.321747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:59.321712    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:59.464203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:59.463245    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:59.959662 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:59.959634    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 19:48:59.960058 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:48:59.959704    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 19:49:00.363070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.362979    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:00.619608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.619499    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:00.653304 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.653272    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"]
Nov 03 19:49:00.659759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.659731    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 19:49:00.664617 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.664408    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-b9cbea-dockercfg-vntgd\""
Nov 03 19:49:00.666751 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.664983    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-b9cbea\""
Nov 03 19:49:00.672734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.672695    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"]
Nov 03 19:49:00.703299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.703264    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"]
Nov 03 19:49:00.710946 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.710904    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 19:49:00.713323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.713299    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"]
Nov 03 19:49:00.865900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.817684    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:00.865900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.822059    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2\" (UID: \"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 19:49:00.865900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.822128    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcstd\" (UniqueName: \"kubernetes.io/projected/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-kube-api-access-qcstd\") pod \"payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2\" (UID: \"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 19:49:00.865900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.822255    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh99f\" (UniqueName: \"kubernetes.io/projected/e9f44648-77e8-408c-9193-9beb63991053-kube-api-access-fh99f\") pod \"payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb\" (UID: \"e9f44648-77e8-408c-9193-9beb63991053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 19:49:00.865900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.822339    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f44648-77e8-408c-9193-9beb63991053-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb\" (UID: \"e9f44648-77e8-408c-9193-9beb63991053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 19:49:01.010254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.954541    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2\" (UID: \"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 19:49:01.010254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.954613    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcstd\" (UniqueName: \"kubernetes.io/projected/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-kube-api-access-qcstd\") pod \"payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2\" (UID: \"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 19:49:01.010254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.954726    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh99f\" (UniqueName: \"kubernetes.io/projected/e9f44648-77e8-408c-9193-9beb63991053-kube-api-access-fh99f\") pod \"payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb\" (UID: \"e9f44648-77e8-408c-9193-9beb63991053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 19:49:01.010254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:00.954814    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f44648-77e8-408c-9193-9beb63991053-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb\" (UID: \"e9f44648-77e8-408c-9193-9beb63991053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 19:49:01.120193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.120124    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh99f\" (UniqueName: \"kubernetes.io/projected/e9f44648-77e8-408c-9193-9beb63991053-kube-api-access-fh99f\") pod \"payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb\" (UID: \"e9f44648-77e8-408c-9193-9beb63991053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 19:49:01.139164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.139124    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcstd\" (UniqueName: \"kubernetes.io/projected/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-kube-api-access-qcstd\") pod \"payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2\" (UID: \"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 19:49:01.144460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.143447    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2\" (UID: \"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 19:49:01.157383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.154247    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"]
Nov 03 19:49:01.163362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.163316    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f44648-77e8-408c-9193-9beb63991053-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb\" (UID: \"e9f44648-77e8-408c-9193-9beb63991053\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 19:49:01.174315 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.164520    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9cd27fa-nhkzs"]
Nov 03 19:49:01.175971 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.175932    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 19:49:01.177239 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.176898    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"]
Nov 03 19:49:01.186766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.186743    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaefc7b4-v54zq"]
Nov 03 19:49:01.202259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.202235    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 19:49:01.285393 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.285213    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 19:49:01.421478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.421141    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:01.490509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.486685    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"]
Nov 03 19:49:01.588406 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:49:01.588375    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f44648_77e8_408c_9193_9beb63991053.slice/crio-4e5e336547ddb3b63dd474c6c63be2a76b6b138221eb7e8208f4d25917194cff WatchSource:0}: Error finding container 4e5e336547ddb3b63dd474c6c63be2a76b6b138221eb7e8208f4d25917194cff: Status 404 returned error can't find the container with id 4e5e336547ddb3b63dd474c6c63be2a76b6b138221eb7e8208f4d25917194cff
Nov 03 19:49:01.591028 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:01.591006    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"]
Nov 03 19:49:02.180201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.171223    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:02.291430 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.291394    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb" event={"ID":"e9f44648-77e8-408c-9193-9beb63991053","Type":"ContainerStarted","Data":"6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647"}
Nov 03 19:49:02.291430 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.291459    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb" event={"ID":"e9f44648-77e8-408c-9193-9beb63991053","Type":"ContainerStarted","Data":"4e5e336547ddb3b63dd474c6c63be2a76b6b138221eb7e8208f4d25917194cff"}
Nov 03 19:49:02.291703 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.291512    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 19:49:02.296747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.296713    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2" event={"ID":"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c","Type":"ContainerStarted","Data":"a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3"}
Nov 03 19:49:02.296882 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.296787    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2" event={"ID":"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c","Type":"ContainerStarted","Data":"fac4913e8f0b1acbee53d44b5779a9f584a4c4f548d91387f4ed37babf47612e"}
Nov 03 19:49:02.296882 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.296816    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 19:49:02.315236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.308468    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb" podStartSLOduration=2.308448242 podStartE2EDuration="2.308448242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:02.305628944 +0000 UTC m=+1055.575752239" watchObservedRunningTime="2026-11-03 19:49:02.308448242 +0000 UTC m=+1055.578571539"
Nov 03 19:49:02.402241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.402161    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2" podStartSLOduration=2.40213667 podStartE2EDuration="2.40213667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:02.320897027 +0000 UTC m=+1055.591020324" watchObservedRunningTime="2026-11-03 19:49:02.40213667 +0000 UTC m=+1055.672259965"
Nov 03 19:49:02.861347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.861317    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eae4dea-9ca7-427b-90e2-683861bba5c1" path="/var/lib/kubelet/pods/1eae4dea-9ca7-427b-90e2-683861bba5c1/volumes"
Nov 03 19:49:02.861858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.861844    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b" path="/var/lib/kubelet/pods/b62e9b3d-e0ed-4fbc-9a2c-a22fdc9ba84b/volumes"
Nov 03 19:49:02.966299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:02.958109    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:03.663205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:03.660742    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:04.421650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:04.417701    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:04.544044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:04.544013    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:05.015346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:05.015314    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:05.379278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:05.336933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:05.755918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:05.755757    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:05.894665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:05.894613    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:06.187167 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.187134    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:06.351285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.349066    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:06.496929 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.496830    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:06.739485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.739032    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"]
Nov 03 19:49:06.755647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.755567    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:06.760455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.760426    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"]
Nov 03 19:49:06.760752 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.760729    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-b9cbea-serving-cert\""
Nov 03 19:49:06.817583 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.817550    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:06.936198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.927121    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4t8\" (UniqueName: \"kubernetes.io/projected/c08e67b7-ff2e-4845-9287-acad83fb619a-kube-api-access-mx4t8\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:06.936198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.927250    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:06.936198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.927324    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:06.936198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.927416    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08e67b7-ff2e-4845-9287-acad83fb619a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:06.963193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:06.952647    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:07.050245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:07.048711    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:07.050245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:07.048828    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:07.050245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:07.048939    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08e67b7-ff2e-4845-9287-acad83fb619a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:07.050245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:07.049313    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4t8\" (UniqueName: \"kubernetes.io/projected/c08e67b7-ff2e-4845-9287-acad83fb619a-kube-api-access-mx4t8\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:07.075538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:07.075158    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08e67b7-ff2e-4845-9287-acad83fb619a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:07.077541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:07.076877    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:07.083986 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:07.083947    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:07.084149 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:07.084129    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4t8\" (UniqueName: \"kubernetes.io/projected/c08e67b7-ff2e-4845-9287-acad83fb619a-kube-api-access-mx4t8\") pod \"maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:07.390081 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:07.388815    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:07.843571 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:07.843532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:07.869552 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:49:07.869516    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08e67b7_ff2e_4845_9287_acad83fb619a.slice/crio-bb6d38a85f826c71e27ed54e2ab08130862381c2d59d41456c61041aaee8e4b9 WatchSource:0}: Error finding container bb6d38a85f826c71e27ed54e2ab08130862381c2d59d41456c61041aaee8e4b9: Status 404 returned error can't find the container with id bb6d38a85f826c71e27ed54e2ab08130862381c2d59d41456c61041aaee8e4b9
Nov 03 19:49:07.871441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:07.871415    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"]
Nov 03 19:49:08.534899 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:08.534865    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm" event={"ID":"c08e67b7-ff2e-4845-9287-acad83fb619a","Type":"ContainerStarted","Data":"68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5"}
Nov 03 19:49:08.535379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:08.534918    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm" event={"ID":"c08e67b7-ff2e-4845-9287-acad83fb619a","Type":"ContainerStarted","Data":"bb6d38a85f826c71e27ed54e2ab08130862381c2d59d41456c61041aaee8e4b9"}
Nov 03 19:49:08.535379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:08.534998    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:08.556503 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:08.556448    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm" podStartSLOduration=2.071044876 podStartE2EDuration="2.556431982s" totalImagesPullingTime="485.387106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:08.549458016 +0000 UTC m=+1061.819581313" watchObservedRunningTime="2026-11-03 19:49:08.556431982 +0000 UTC m=+1061.826555276"
Nov 03 19:49:09.081529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:09.064717    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:09.081529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:09.070245    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"]
Nov 03 19:49:09.117196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:09.109248    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"]
Nov 03 19:49:09.128309 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:09.126244    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-326dc398-hfjt8"]
Nov 03 19:49:09.134194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:09.132344    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7868645a-mhd42"]
Nov 03 19:49:09.633203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:09.633157    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:09.765371 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:09.765341    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:09.908008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:09.907881    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:10.714924 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.714866    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:10.866257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.866216    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae49a5f-2faf-4388-b697-8d6dcccb4f97" path="/var/lib/kubelet/pods/2ae49a5f-2faf-4388-b697-8d6dcccb4f97/volumes"
Nov 03 19:49:10.866990 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.866962    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff74c7d-b1eb-4ed2-b150-b058cc95d542" path="/var/lib/kubelet/pods/cff74c7d-b1eb-4ed2-b150-b058cc95d542/volumes"
Nov 03 19:49:10.937530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.937492    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"]
Nov 03 19:49:10.943923 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.943891    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 19:49:10.947841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.947809    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-2d78ea\""
Nov 03 19:49:10.947999 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.947957    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-2d78ea-dockercfg-jnggb\""
Nov 03 19:49:10.949834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.949810    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"]
Nov 03 19:49:10.958690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.958661    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:10.979308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.979207    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"]
Nov 03 19:49:10.984884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.984853    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 19:49:10.990388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.990355    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"]
Nov 03 19:49:10.992969 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.992937    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m98t\" (UniqueName: \"kubernetes.io/projected/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-kube-api-access-2m98t\") pod \"payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm\" (UID: \"746b1f72-ee9f-4a64-bf2f-7db1ae805aef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 19:49:10.993238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:10.993161    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm\" (UID: \"746b1f72-ee9f-4a64-bf2f-7db1ae805aef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 19:49:11.173194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.158610    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:11.235256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.099481    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m98t\" (UniqueName: \"kubernetes.io/projected/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-kube-api-access-2m98t\") pod \"payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm\" (UID: \"746b1f72-ee9f-4a64-bf2f-7db1ae805aef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 19:49:11.235256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.221914    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm\" (UID: \"746b1f72-ee9f-4a64-bf2f-7db1ae805aef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 19:49:11.235256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.221959    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcm9x\" (UniqueName: \"kubernetes.io/projected/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-kube-api-access-qcm9x\") pod \"payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4\" (UID: \"3e71a759-fb64-494f-8e4d-6aa7a427f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 19:49:11.235256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.222096    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4\" (UID: \"3e71a759-fb64-494f-8e4d-6aa7a427f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 19:49:11.235256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.223437    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm\" (UID: \"746b1f72-ee9f-4a64-bf2f-7db1ae805aef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 19:49:11.291210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.286006    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m98t\" (UniqueName: \"kubernetes.io/projected/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-kube-api-access-2m98t\") pod \"payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm\" (UID: \"746b1f72-ee9f-4a64-bf2f-7db1ae805aef\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 19:49:11.309222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.292710    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 19:49:11.326593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.324634    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcm9x\" (UniqueName: \"kubernetes.io/projected/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-kube-api-access-qcm9x\") pod \"payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4\" (UID: \"3e71a759-fb64-494f-8e4d-6aa7a427f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 19:49:11.326593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.324938    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4\" (UID: \"3e71a759-fb64-494f-8e4d-6aa7a427f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 19:49:11.326593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.326437    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4\" (UID: \"3e71a759-fb64-494f-8e4d-6aa7a427f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 19:49:11.410195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.381296    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcm9x\" (UniqueName: \"kubernetes.io/projected/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-kube-api-access-qcm9x\") pod \"payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4\" (UID: \"3e71a759-fb64-494f-8e4d-6aa7a427f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 19:49:11.605897 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.605862    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 19:49:11.736319 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:49:11.736285    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746b1f72_ee9f_4a64_bf2f_7db1ae805aef.slice/crio-3b144541a973361fbc3e21e133b5c0c3bb38285a6812c50798cd125581d8c009 WatchSource:0}: Error finding container 3b144541a973361fbc3e21e133b5c0c3bb38285a6812c50798cd125581d8c009: Status 404 returned error can't find the container with id 3b144541a973361fbc3e21e133b5c0c3bb38285a6812c50798cd125581d8c009
Nov 03 19:49:11.772263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.750823    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"]
Nov 03 19:49:11.813211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.789487    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:11.908760 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:11.908721    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"]
Nov 03 19:49:12.490850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:12.490809    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:12.747848 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:12.747738    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm" event={"ID":"746b1f72-ee9f-4a64-bf2f-7db1ae805aef","Type":"ContainerStarted","Data":"2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8"}
Nov 03 19:49:12.752238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:12.752207    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 19:49:12.752504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:12.752475    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm" event={"ID":"746b1f72-ee9f-4a64-bf2f-7db1ae805aef","Type":"ContainerStarted","Data":"3b144541a973361fbc3e21e133b5c0c3bb38285a6812c50798cd125581d8c009"}
Nov 03 19:49:12.753565 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:12.753535    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4" event={"ID":"3e71a759-fb64-494f-8e4d-6aa7a427f6e3","Type":"ContainerStarted","Data":"29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7"}
Nov 03 19:49:12.754009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:12.753974    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4" event={"ID":"3e71a759-fb64-494f-8e4d-6aa7a427f6e3","Type":"ContainerStarted","Data":"fd3c9708a21210d962c4729de76ae33f89fe79d7d46c97f9e0e732eea37ba68d"}
Nov 03 19:49:12.755684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:12.755649    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 19:49:12.757579 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:12.757553    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:12.789845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:12.789772    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm" podStartSLOduration=2.789753508 podStartE2EDuration="2.789753508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:12.777202497 +0000 UTC m=+1066.047325793" watchObservedRunningTime="2026-11-03 19:49:12.789753508 +0000 UTC m=+1066.059876805"
Nov 03 19:49:12.792964 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:12.792918    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4" podStartSLOduration=2.79290459 podStartE2EDuration="2.79290459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:12.790802117 +0000 UTC m=+1066.060925414" watchObservedRunningTime="2026-11-03 19:49:12.79290459 +0000 UTC m=+1066.063027886"
Nov 03 19:49:13.265953 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:13.264073    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:13.321295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:13.321254    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 19:49:13.323237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:13.323199    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 19:49:13.868134 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:13.868021    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:14.023123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:14.023088    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"]
Nov 03 19:49:14.028799 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:14.028768    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f50d8f7-hpb9t"]
Nov 03 19:49:14.550654 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:14.550623    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 19:49:14.605677 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:14.605629    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:14.861036 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:14.860960    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7947c08-1f0f-49d8-a23b-53dd6d666355" path="/var/lib/kubelet/pods/e7947c08-1f0f-49d8-a23b-53dd6d666355/volumes"
Nov 03 19:49:15.245630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:15.245582    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:15.732629 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:15.732593    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:16.038669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:16.038522    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:16.962197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:16.961358    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:17.726664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:17.726618    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:17.873200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:17.867164    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:18.221199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:18.215832    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:18.669870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:18.669816    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:19.392080 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:19.392043    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:19.538092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:19.538057    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:20.147457 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:20.143637    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:20.607553 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:49:20.607475    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:49:20.852721 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:20.852672    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:21.063527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:21.063493    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"]
Nov 03 19:49:21.075640 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:21.075606    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfa4f37b-fgnbx"]
Nov 03 19:49:21.524405 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:21.524368    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:22.023683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:22.023640    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"]
Nov 03 19:49:22.029651 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:22.029621    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3675c37f-vssgq"]
Nov 03 19:49:22.216635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:22.216600    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:22.403257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:22.400592    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:22.794735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:22.794644    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:23.098251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:23.080812    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:23.178188 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:23.178126    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1081e45b-8763-4e99-acb3-b94a192fa8a8" path="/var/lib/kubelet/pods/1081e45b-8763-4e99-acb3-b94a192fa8a8/volumes"
Nov 03 19:49:23.188206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:23.179411    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6" path="/var/lib/kubelet/pods/3bc6006b-a4d1-4f1d-b761-ad0fdc5a52e6/volumes"
Nov 03 19:49:23.667013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:23.666974    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:23.815733 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:23.815677    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:24.338800 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:24.338747    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:24.789268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:24.789237    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 19:49:24.790893 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:24.790867    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 19:49:25.162199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:25.159924    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:26.005089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.005042    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:26.516533 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.516480    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"]
Nov 03 19:49:26.550326 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.548948    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"]
Nov 03 19:49:26.550326 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.549088    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.581045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.580049    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4dm\" (UniqueName: \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-kube-api-access-2h4dm\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.581045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.580118    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.581045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.580259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.581045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.580312    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.581045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.580433    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.581045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.580466    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.581045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.580566    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.581045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.580648    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.581045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.580699    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.682364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.682316    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.682364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.682366    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.682584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.682461    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.682584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.682534    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.682584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.682578    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.682755 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.682736    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4dm\" (UniqueName: \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-kube-api-access-2h4dm\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.682791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.682770    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.682791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.682780    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.683058 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.683039    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.683324 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.683303    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.683390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.683373    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.683742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.683721    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.684317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.684288    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.684608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.684549    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.686297 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.686275    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.686634 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.686610    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.712300 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.712260    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4dm\" (UniqueName: \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-kube-api-access-2h4dm\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.715805 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.715768    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:26.816467 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.816391    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:26.875500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.875379    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:27.006140 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:26.999755    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:27.579224 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:49:27.578789    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0edc976_7bb9_4cdf_82d4_cdd3ba9c92fd.slice/crio-fecf34ff898fdbc877e1327ddb5417c490e42109b13323e960bc3d0ceac7c0bf WatchSource:0}: Error finding container fecf34ff898fdbc877e1327ddb5417c490e42109b13323e960bc3d0ceac7c0bf: Status 404 returned error can't find the container with id fecf34ff898fdbc877e1327ddb5417c490e42109b13323e960bc3d0ceac7c0bf
Nov 03 19:49:27.580420 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:27.580395    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"]
Nov 03 19:49:27.582416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:27.582337    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:27.582534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:27.582433    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:27.582534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:27.582480    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:28.185931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:28.185709    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:28.344870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:28.344781    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" event={"ID":"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd","Type":"ContainerStarted","Data":"1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9"}
Nov 03 19:49:28.344870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:28.344844    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" event={"ID":"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd","Type":"ContainerStarted","Data":"fecf34ff898fdbc877e1327ddb5417c490e42109b13323e960bc3d0ceac7c0bf"}
Nov 03 19:49:28.371539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:28.371467    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" podStartSLOduration=2.3714452010000002 podStartE2EDuration="2.371445201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:28.365375802 +0000 UTC m=+1081.635499098" watchObservedRunningTime="2026-11-03 19:49:28.371445201 +0000 UTC m=+1081.641568677"
Nov 03 19:49:28.883255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:28.883222    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 19:49:28.883599 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:28.883570    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 19:49:28.886463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:28.886434    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:29.069267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.069226    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"]
Nov 03 19:49:29.075631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.075593    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df208d50-wg264"]
Nov 03 19:49:29.108789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.108532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:29.175956 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.175264    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"]
Nov 03 19:49:29.202393 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.201856    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"]
Nov 03 19:49:29.202393 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.201918    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"]
Nov 03 19:49:29.202393 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.202016    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:49:29.207268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.206920    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a536f35b\""
Nov 03 19:49:29.207607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.207582    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a536f35b-dockercfg-4hq7c\""
Nov 03 19:49:29.222063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.222025    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"]
Nov 03 19:49:29.222212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.222161    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:49:29.366195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.343637    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:29.539268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.537445    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndzz\" (UniqueName: \"kubernetes.io/projected/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-kube-api-access-jndzz\") pod \"payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr\" (UID: \"ea449c7e-8da9-4c54-a81a-dfcdff11cef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:49:29.539268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.537570    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2k8h\" (UniqueName: \"kubernetes.io/projected/df962d0b-648a-4b8c-b894-08ae29ac036e-kube-api-access-m2k8h\") pod \"payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8\" (UID: \"df962d0b-648a-4b8c-b894-08ae29ac036e\") " pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:49:29.539268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.537655    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr\" (UID: \"ea449c7e-8da9-4c54-a81a-dfcdff11cef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:49:29.539268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.537968    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df962d0b-648a-4b8c-b894-08ae29ac036e-plugins-config-volume\") pod \"payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8\" (UID: \"df962d0b-648a-4b8c-b894-08ae29ac036e\") " pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:49:29.645282 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.612410    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:29.671218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.666742    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df962d0b-648a-4b8c-b894-08ae29ac036e-plugins-config-volume\") pod \"payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8\" (UID: \"df962d0b-648a-4b8c-b894-08ae29ac036e\") " pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:49:29.671218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.666867    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndzz\" (UniqueName: \"kubernetes.io/projected/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-kube-api-access-jndzz\") pod \"payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr\" (UID: \"ea449c7e-8da9-4c54-a81a-dfcdff11cef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:49:29.671218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.666976    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2k8h\" (UniqueName: \"kubernetes.io/projected/df962d0b-648a-4b8c-b894-08ae29ac036e-kube-api-access-m2k8h\") pod \"payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8\" (UID: \"df962d0b-648a-4b8c-b894-08ae29ac036e\") " pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:49:29.671218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.667073    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr\" (UID: \"ea449c7e-8da9-4c54-a81a-dfcdff11cef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:49:29.671218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.668351    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr\" (UID: \"ea449c7e-8da9-4c54-a81a-dfcdff11cef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:49:29.671218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.669138    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df962d0b-648a-4b8c-b894-08ae29ac036e-plugins-config-volume\") pod \"payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8\" (UID: \"df962d0b-648a-4b8c-b894-08ae29ac036e\") " pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:49:29.777910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.774493    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:29.792255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.788999    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndzz\" (UniqueName: \"kubernetes.io/projected/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-kube-api-access-jndzz\") pod \"payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr\" (UID: \"ea449c7e-8da9-4c54-a81a-dfcdff11cef9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:49:29.792255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.789122    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:49:29.792255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.789190    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:29.812361 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.811660    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2k8h\" (UniqueName: \"kubernetes.io/projected/df962d0b-648a-4b8c-b894-08ae29ac036e-kube-api-access-m2k8h\") pod \"payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8\" (UID: \"df962d0b-648a-4b8c-b894-08ae29ac036e\") " pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:49:29.880597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.880562    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 19:49:29.880746 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.880616    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 19:49:29.883548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.883288    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:49:29.979655 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:29.979306    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:49:30.062250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.061784    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:30.231010 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.230868    4531 scope.go:122] "RemoveContainer" containerID="feb90ff1a2e8944d4085db1d9ca194a87a1af834bfe01ce3cb4008d7125025d1"
Nov 03 19:49:30.438276 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.438243    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"]
Nov 03 19:49:30.496970 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.496942    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7096-98e19a"]
Nov 03 19:49:30.564520 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.564486    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-98e19a"]
Nov 03 19:49:30.564642 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.564617    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-98e19a"
Nov 03 19:49:30.568690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.568633    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bntlr\""
Nov 03 19:49:30.574052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.572654    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"]
Nov 03 19:49:30.589094 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.589064    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pg4f\" (UniqueName: \"kubernetes.io/projected/75e7b472-beb2-45a9-a12b-48d9105e3795-kube-api-access-5pg4f\") pod \"test-curl-7096-98e19a\" (UID: \"75e7b472-beb2-45a9-a12b-48d9105e3795\") " pod="opendatahub/test-curl-7096-98e19a"
Nov 03 19:49:30.691675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.691508    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pg4f\" (UniqueName: \"kubernetes.io/projected/75e7b472-beb2-45a9-a12b-48d9105e3795-kube-api-access-5pg4f\") pod \"test-curl-7096-98e19a\" (UID: \"75e7b472-beb2-45a9-a12b-48d9105e3795\") " pod="opendatahub/test-curl-7096-98e19a"
Nov 03 19:49:30.711776 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.711505    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pg4f\" (UniqueName: \"kubernetes.io/projected/75e7b472-beb2-45a9-a12b-48d9105e3795-kube-api-access-5pg4f\") pod \"test-curl-7096-98e19a\" (UID: \"75e7b472-beb2-45a9-a12b-48d9105e3795\") " pod="opendatahub/test-curl-7096-98e19a"
Nov 03 19:49:30.740794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.739411    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8" event={"ID":"df962d0b-648a-4b8c-b894-08ae29ac036e","Type":"ContainerStarted","Data":"84109e8e47f0c9289955ca6fd7e2d233f3d0d7447f85bfd6528bcb58ecf95f96"}
Nov 03 19:49:30.740794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.739482    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8" event={"ID":"df962d0b-648a-4b8c-b894-08ae29ac036e","Type":"ContainerStarted","Data":"c2a018a03164f06bd49b7db264ec8f74656e39117df2722f734aaa277a6b3af0"}
Nov 03 19:49:30.741192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.741105    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:49:30.745517 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.745489    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr" event={"ID":"ea449c7e-8da9-4c54-a81a-dfcdff11cef9","Type":"ContainerStarted","Data":"247f5f70a70e5546f6946ca63f64746e68ebe7eaf77046c3652dc0c387b90f38"}
Nov 03 19:49:30.745766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.745749    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:49:30.745923 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.745906    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr" event={"ID":"ea449c7e-8da9-4c54-a81a-dfcdff11cef9","Type":"ContainerStarted","Data":"5e1a0dba56f89088ef6d6ec44ed0ae4a1f63376f07b57e5d5022bbcdf17d217a"}
Nov 03 19:49:30.781472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.781424    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8" podStartSLOduration=1.781409215 podStartE2EDuration="1.781409215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:30.78041186 +0000 UTC m=+1084.050535156" watchObservedRunningTime="2026-11-03 19:49:30.781409215 +0000 UTC m=+1084.051532510"
Nov 03 19:49:30.797433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.797378    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr" podStartSLOduration=1.797363305 podStartE2EDuration="1.797363305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:30.794782357 +0000 UTC m=+1084.064905653" watchObservedRunningTime="2026-11-03 19:49:30.797363305 +0000 UTC m=+1084.067486626"
Nov 03 19:49:30.871390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.871311    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229" path="/var/lib/kubelet/pods/53c35dcf-fe30-4e9b-86fd-ca5d0a7cf229/volumes"
Nov 03 19:49:30.876758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.876730    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 19:49:30.876898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.876780    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 19:49:30.923999 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:30.923513    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-98e19a"
Nov 03 19:49:31.110631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.081565    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:49:31.110631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.081651    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:31.184258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.183990    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:31.531833 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:49:31.531802    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75e7b472_beb2_45a9_a12b_48d9105e3795.slice/crio-657d60fe7d5ab81c20fd6a1b9cdaec39c4e6455658d2d1719de3b3002c4b684a WatchSource:0}: Error finding container 657d60fe7d5ab81c20fd6a1b9cdaec39c4e6455658d2d1719de3b3002c4b684a: Status 404 returned error can't find the container with id 657d60fe7d5ab81c20fd6a1b9cdaec39c4e6455658d2d1719de3b3002c4b684a
Nov 03 19:49:31.576244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.533647    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-98e19a"]
Nov 03 19:49:31.773214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.772988    4531 generic.go:356] "Generic (PLEG): container finished" podID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerID="a9750e8a6fd96da631ad0f22a2e3f730b213834905cd3fe34b63fab28b7e2e91" exitCode=0
Nov 03 19:49:31.773214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.773060    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" event={"ID":"385aa4c7-24cc-4a11-a69b-e572797c115f","Type":"ContainerDied","Data":"a9750e8a6fd96da631ad0f22a2e3f730b213834905cd3fe34b63fab28b7e2e91"}
Nov 03 19:49:31.773661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.773642    4531 scope.go:122] "RemoveContainer" containerID="a9750e8a6fd96da631ad0f22a2e3f730b213834905cd3fe34b63fab28b7e2e91"
Nov 03 19:49:31.774604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.774583    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-98e19a" event={"ID":"75e7b472-beb2-45a9-a12b-48d9105e3795","Type":"ContainerStarted","Data":"657d60fe7d5ab81c20fd6a1b9cdaec39c4e6455658d2d1719de3b3002c4b684a"}
Nov 03 19:49:31.775673 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.775637    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:31.775789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.775730    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:31.775789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.775778    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:31.776577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.776558    4531 generic.go:356] "Generic (PLEG): container finished" podID="b63cceab-6dee-4689-8767-9a472d99b541" containerID="1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577" exitCode=0
Nov 03 19:49:31.776680 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.776661    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" event={"ID":"b63cceab-6dee-4689-8767-9a472d99b541","Type":"ContainerDied","Data":"1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577"}
Nov 03 19:49:31.777275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.777257    4531 scope.go:122] "RemoveContainer" containerID="1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577"
Nov 03 19:49:31.779759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.779731    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:31.779832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.779793    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:31.779832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.779827    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:31.877257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.875987    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 19:49:31.877257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.876053    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 19:49:31.959866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:31.959493    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:32.661199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:32.655116    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:32.830198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:32.824308    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" event={"ID":"385aa4c7-24cc-4a11-a69b-e572797c115f","Type":"ContainerStarted","Data":"9155eb75af231eefdc7b451ab1c9a3624fabd9153b7e02d1e80d4a204d90e4c5"}
Nov 03 19:49:32.890542 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:32.890504    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 19:49:32.890836 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:32.890801    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 19:49:32.969691 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:32.959486    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" event={"ID":"b63cceab-6dee-4689-8767-9a472d99b541","Type":"ContainerStarted","Data":"5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444"}
Nov 03 19:49:33.074199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:33.066187    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:49:33.074199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:33.066310    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 19:49:33.074199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:33.066355    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 19:49:33.074921 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:33.074892    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:49:33.115197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:33.088302    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 19:49:33.115197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:33.088366    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 19:49:33.376330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:33.374353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:33.912505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:33.911069    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 19:49:33.912505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:33.911130    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 19:49:34.055616 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:34.055575    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 19:49:34.062358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:34.061893    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 19:49:34.073606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:34.073568    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 19:49:34.073869 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:34.073834    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 19:49:34.122907 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:34.122865    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:34.914192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:34.911216    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 19:49:34.914192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:34.911280    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 19:49:34.943263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:34.928086    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:35.087661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:35.087625    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 19:49:35.087995 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:35.087947    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 19:49:35.128818 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:35.107561    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 19:49:35.128818 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:35.107621    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 19:49:35.687197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:35.676036    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:35.902061 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:35.902021    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:35.902902 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:35.902833    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:35.904507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:35.904382    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 19:49:36.097252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:36.068354    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 19:49:36.097252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:36.068438    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 19:49:36.112403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:36.112355    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 19:49:36.112706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:36.112663    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 19:49:36.320860 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:36.320821    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:36.470930 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:36.470889    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:36.976495 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:36.976452    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:37.100222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:37.081033    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 19:49:37.100222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:37.081139    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 19:49:37.160233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:37.160197    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 19:49:37.160512 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:37.160481    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 19:49:37.242248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:37.237160    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:38.003750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:38.003712    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:38.095234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:38.087520    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 19:49:38.095234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:38.087594    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 19:49:38.095660 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:38.095629    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 19:49:38.095861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:38.095828    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 19:49:38.658199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:38.644723    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:39.054675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:39.054573    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 19:49:39.059205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:39.054917    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 19:49:39.089779 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:39.089742    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 19:49:39.090555 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:39.090510    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 19:49:39.351254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:39.349652    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:40.060375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:40.060335    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:49:40.061432 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:40.061409    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:49:40.062249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:40.062230    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:49:40.075505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:40.075416    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:40.172506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:40.172462    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:49:40.174909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:40.174872    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:49:40.186249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:40.185575    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 19:49:40.731889 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:40.731776    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:41.554194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:41.552872    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:41.583055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:41.583002    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-98e19a" event={"ID":"75e7b472-beb2-45a9-a12b-48d9105e3795","Type":"ContainerStarted","Data":"21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d"}
Nov 03 19:49:41.639589 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:41.639525    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7096-98e19a" podStartSLOduration=2.714320621 podStartE2EDuration="11.639502359s" totalImagesPullingTime="8.925181738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:41.614391005 +0000 UTC m=+1094.884514301" watchObservedRunningTime="2026-11-03 19:49:41.639502359 +0000 UTC m=+1094.909625652"
Nov 03 19:49:41.780668 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:41.780638    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:49:42.274196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.263861    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:42.313353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.313314    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7096-98e19a"]
Nov 03 19:49:42.350481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.350444    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7096-98e19a"]
Nov 03 19:49:42.400007 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.398724    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7096-f2ca06"]
Nov 03 19:49:42.400007 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.399997    4531 cpu_manager.go:414] "containerMap: removing container" podUID="75e7b472-beb2-45a9-a12b-48d9105e3795" containerName="test-curl-7096-98e19a"
Nov 03 19:49:42.400292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.400020    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e7b472-beb2-45a9-a12b-48d9105e3795" containerName="test-curl-7096-98e19a"
Nov 03 19:49:42.400359 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.400294    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e7b472-beb2-45a9-a12b-48d9105e3795" containerName="test-curl-7096-98e19a"
Nov 03 19:49:42.409202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.404903    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-f2ca06"
Nov 03 19:49:42.419570 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.419535    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-f2ca06"]
Nov 03 19:49:42.498310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.498266    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6tfp\" (UniqueName: \"kubernetes.io/projected/58298239-da8a-4443-a7a8-e3ceb9d7f3a8-kube-api-access-q6tfp\") pod \"test-curl-7096-f2ca06\" (UID: \"58298239-da8a-4443-a7a8-e3ceb9d7f3a8\") " pod="opendatahub/test-curl-7096-f2ca06"
Nov 03 19:49:42.599985 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.599897    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6tfp\" (UniqueName: \"kubernetes.io/projected/58298239-da8a-4443-a7a8-e3ceb9d7f3a8-kube-api-access-q6tfp\") pod \"test-curl-7096-f2ca06\" (UID: \"58298239-da8a-4443-a7a8-e3ceb9d7f3a8\") " pod="opendatahub/test-curl-7096-f2ca06"
Nov 03 19:49:42.607300 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.607276    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6tfp\" (UniqueName: \"kubernetes.io/projected/58298239-da8a-4443-a7a8-e3ceb9d7f3a8-kube-api-access-q6tfp\") pod \"test-curl-7096-f2ca06\" (UID: \"58298239-da8a-4443-a7a8-e3ceb9d7f3a8\") " pod="opendatahub/test-curl-7096-f2ca06"
Nov 03 19:49:42.730568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.730527    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-f2ca06"
Nov 03 19:49:42.896708 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.896598    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:42.973281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:42.964048    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:49:43.049688 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:49:43.049660    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58298239_da8a_4443_a7a8_e3ceb9d7f3a8.slice/crio-af4bbd24762214b5f64e36ba2e60ae06fd3da29d33bcf9f0cfda47d8d5f23a67 WatchSource:0}: Error finding container af4bbd24762214b5f64e36ba2e60ae06fd3da29d33bcf9f0cfda47d8d5f23a67: Status 404 returned error can't find the container with id af4bbd24762214b5f64e36ba2e60ae06fd3da29d33bcf9f0cfda47d8d5f23a67
Nov 03 19:49:43.057708 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:43.057681    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-f2ca06"]
Nov 03 19:49:43.589337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:43.581646    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:43.656086 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:43.656050    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-f2ca06" event={"ID":"58298239-da8a-4443-a7a8-e3ceb9d7f3a8","Type":"ContainerStarted","Data":"0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026"}
Nov 03 19:49:43.656622 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:43.656096    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-f2ca06" event={"ID":"58298239-da8a-4443-a7a8-e3ceb9d7f3a8","Type":"ContainerStarted","Data":"af4bbd24762214b5f64e36ba2e60ae06fd3da29d33bcf9f0cfda47d8d5f23a67"}
Nov 03 19:49:43.656622 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:43.656548    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7096-98e19a" podUID="75e7b472-beb2-45a9-a12b-48d9105e3795" containerName="test-curl-7096-98e19a" containerID="cri-o://21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d" gracePeriod=2
Nov 03 19:49:43.687395 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:43.687337    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7096-f2ca06" podStartSLOduration=1.687320731 podStartE2EDuration="1.687320731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:43.68044626 +0000 UTC m=+1096.950569558" watchObservedRunningTime="2026-11-03 19:49:43.687320731 +0000 UTC m=+1096.957444020"
Nov 03 19:49:44.276195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.276041    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:44.384096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.384060    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7096-f2ca06"]
Nov 03 19:49:44.392990 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.392947    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7096-f2ca06"]
Nov 03 19:49:44.509122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.509086    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7096-95b4b7"]
Nov 03 19:49:44.510378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.510349    4531 cpu_manager.go:414] "containerMap: removing container" podUID="58298239-da8a-4443-a7a8-e3ceb9d7f3a8" containerName="test-curl-7096-f2ca06"
Nov 03 19:49:44.510378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.510377    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58298239-da8a-4443-a7a8-e3ceb9d7f3a8" containerName="test-curl-7096-f2ca06"
Nov 03 19:49:44.510603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.510568    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="58298239-da8a-4443-a7a8-e3ceb9d7f3a8" containerName="test-curl-7096-f2ca06"
Nov 03 19:49:44.515226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.515158    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-95b4b7"
Nov 03 19:49:44.517134 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.517106    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-95b4b7"]
Nov 03 19:49:44.690496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.690460    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjkt\" (UniqueName: \"kubernetes.io/projected/4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f-kube-api-access-qsjkt\") pod \"test-curl-7096-95b4b7\" (UID: \"4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f\") " pod="opendatahub/test-curl-7096-95b4b7"
Nov 03 19:49:44.792673 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.792637    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjkt\" (UniqueName: \"kubernetes.io/projected/4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f-kube-api-access-qsjkt\") pod \"test-curl-7096-95b4b7\" (UID: \"4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f\") " pod="opendatahub/test-curl-7096-95b4b7"
Nov 03 19:49:44.802233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.802203    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjkt\" (UniqueName: \"kubernetes.io/projected/4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f-kube-api-access-qsjkt\") pod \"test-curl-7096-95b4b7\" (UID: \"4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f\") " pod="opendatahub/test-curl-7096-95b4b7"
Nov 03 19:49:44.834269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:44.834225    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-95b4b7"
Nov 03 19:49:45.066122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:45.064531    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:45.145876 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:49:45.145849    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cc1c5d9_d2e1_40a6_b6fc_5ad4b127a85f.slice/crio-995cdc917387405e91f937374aa011f9b44b61d1d3398d4c16275f93144148b4 WatchSource:0}: Error finding container 995cdc917387405e91f937374aa011f9b44b61d1d3398d4c16275f93144148b4: Status 404 returned error can't find the container with id 995cdc917387405e91f937374aa011f9b44b61d1d3398d4c16275f93144148b4
Nov 03 19:49:45.151136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:45.151109    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-95b4b7"]
Nov 03 19:49:45.614361 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:45.614327    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:45.749206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:45.743587    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7096-f2ca06" podUID="58298239-da8a-4443-a7a8-e3ceb9d7f3a8" containerName="test-curl-7096-f2ca06" containerID="cri-o://0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026" gracePeriod=2
Nov 03 19:49:45.749206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:45.744839    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-95b4b7" event={"ID":"4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f","Type":"ContainerStarted","Data":"3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d"}
Nov 03 19:49:45.749206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:45.744881    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-95b4b7" event={"ID":"4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f","Type":"ContainerStarted","Data":"995cdc917387405e91f937374aa011f9b44b61d1d3398d4c16275f93144148b4"}
Nov 03 19:49:45.774586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:45.774535    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7096-95b4b7" podStartSLOduration=1.774520769 podStartE2EDuration="1.774520769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:45.763884055 +0000 UTC m=+1099.034007351" watchObservedRunningTime="2026-11-03 19:49:45.774520769 +0000 UTC m=+1099.044644064"
Nov 03 19:49:45.940012 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:45.939988    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-98e19a"
Nov 03 19:49:46.018397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.018363    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e7b472-beb2-45a9-a12b-48d9105e3795-kube-api-access-5pg4f\" (UniqueName: \"kubernetes.io/projected/75e7b472-beb2-45a9-a12b-48d9105e3795-kube-api-access-5pg4f\") pod \"75e7b472-beb2-45a9-a12b-48d9105e3795\" (UID: \"75e7b472-beb2-45a9-a12b-48d9105e3795\") "
Nov 03 19:49:46.020505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.020482    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e7b472-beb2-45a9-a12b-48d9105e3795-kube-api-access-5pg4f" pod "75e7b472-beb2-45a9-a12b-48d9105e3795" (UID: "75e7b472-beb2-45a9-a12b-48d9105e3795"). InnerVolumeSpecName "kube-api-access-5pg4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:46.120905 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.120858    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pg4f\" (UniqueName: \"kubernetes.io/projected/75e7b472-beb2-45a9-a12b-48d9105e3795-kube-api-access-5pg4f\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:46.315252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.314207    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:46.713133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.713098    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7096-95b4b7"]
Nov 03 19:49:46.719536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.719496    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7096-95b4b7"]
Nov 03 19:49:46.772042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.766315    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:46.787232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.785812    4531 generic.go:356] "Generic (PLEG): container finished" podID="75e7b472-beb2-45a9-a12b-48d9105e3795" containerID="21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d" exitCode=137
Nov 03 19:49:46.787467 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.787321    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-98e19a"
Nov 03 19:49:46.796265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.791378    4531 scope.go:122] "RemoveContainer" containerID="21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d"
Nov 03 19:49:46.881470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.880115    4531 scope.go:122] "RemoveContainer" containerID="21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d"
Nov 03 19:49:46.882419 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:49:46.882386    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d\": container with ID starting with 21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d not found: ID does not exist" containerID="21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d"
Nov 03 19:49:46.882532 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.882431    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d"} err="failed to get container status \"21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d\": rpc error: code = NotFound desc = could not find container \"21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d\": container with ID starting with 21b8cfeaf47552faae34da3fc8a5ab2ed74a17932b6a826242e3637ccce6359d not found: ID does not exist"
Nov 03 19:49:46.891785 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:49:46.891735    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-98e19a\" is forbidden: User \"system:node:ip-10-0-131-234.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-234.us-east-2.compute.internal' and this object" podUID="75e7b472-beb2-45a9-a12b-48d9105e3795" pod="opendatahub/test-curl-7096-98e19a"
Nov 03 19:49:46.891942 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.891809    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e7b472-beb2-45a9-a12b-48d9105e3795" path="/var/lib/kubelet/pods/75e7b472-beb2-45a9-a12b-48d9105e3795/volumes"
Nov 03 19:49:46.892329 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.892303    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7096-1f4900"]
Nov 03 19:49:46.893516 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.893493    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f" containerName="test-curl-7096-95b4b7"
Nov 03 19:49:46.893516 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.893516    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f" containerName="test-curl-7096-95b4b7"
Nov 03 19:49:46.893726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.893714    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f" containerName="test-curl-7096-95b4b7"
Nov 03 19:49:46.900013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.899981    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-1f4900"]
Nov 03 19:49:46.900157 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.900099    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-1f4900"
Nov 03 19:49:46.977225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:46.977130    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvkp\" (UniqueName: \"kubernetes.io/projected/98e72f1a-7282-4d59-9749-ddd4b3f72c24-kube-api-access-swvkp\") pod \"test-curl-7096-1f4900\" (UID: \"98e72f1a-7282-4d59-9749-ddd4b3f72c24\") " pod="opendatahub/test-curl-7096-1f4900"
Nov 03 19:49:47.078785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:47.078748    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvkp\" (UniqueName: \"kubernetes.io/projected/98e72f1a-7282-4d59-9749-ddd4b3f72c24-kube-api-access-swvkp\") pod \"test-curl-7096-1f4900\" (UID: \"98e72f1a-7282-4d59-9749-ddd4b3f72c24\") " pod="opendatahub/test-curl-7096-1f4900"
Nov 03 19:49:47.086226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:47.086201    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvkp\" (UniqueName: \"kubernetes.io/projected/98e72f1a-7282-4d59-9749-ddd4b3f72c24-kube-api-access-swvkp\") pod \"test-curl-7096-1f4900\" (UID: \"98e72f1a-7282-4d59-9749-ddd4b3f72c24\") " pod="opendatahub/test-curl-7096-1f4900"
Nov 03 19:49:47.232454 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:47.232364    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-1f4900"
Nov 03 19:49:47.345550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:47.327663    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:47.526618 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:47.520500    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7096-1f4900"]
Nov 03 19:49:47.804245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:47.804207    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-1f4900" event={"ID":"98e72f1a-7282-4d59-9749-ddd4b3f72c24","Type":"ContainerStarted","Data":"6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3"}
Nov 03 19:49:47.804672 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:47.804262    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7096-1f4900" event={"ID":"98e72f1a-7282-4d59-9749-ddd4b3f72c24","Type":"ContainerStarted","Data":"fa45e5f7b2d02f045b6966af588d1e108aa6cb6a1ff529e6b16707bcb34cb02b"}
Nov 03 19:49:47.806081 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:47.806014    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7096-95b4b7" podUID="4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f" containerName="test-curl-7096-95b4b7" containerID="cri-o://3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d" gracePeriod=2
Nov 03 19:49:47.825506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:47.825432    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7096-1f4900" podStartSLOduration=1.825409773 podStartE2EDuration="1.825409773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:47.817565692 +0000 UTC m=+1101.087688987" watchObservedRunningTime="2026-11-03 19:49:47.825409773 +0000 UTC m=+1101.095533069"
Nov 03 19:49:47.965191 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:47.965150    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-f2ca06"
Nov 03 19:49:48.050264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.045932    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:48.089194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.077356    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58298239-da8a-4443-a7a8-e3ceb9d7f3a8-kube-api-access-q6tfp\" (UniqueName: \"kubernetes.io/projected/58298239-da8a-4443-a7a8-e3ceb9d7f3a8-kube-api-access-q6tfp\") pod \"58298239-da8a-4443-a7a8-e3ceb9d7f3a8\" (UID: \"58298239-da8a-4443-a7a8-e3ceb9d7f3a8\") "
Nov 03 19:49:48.149914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.149382    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58298239-da8a-4443-a7a8-e3ceb9d7f3a8-kube-api-access-q6tfp" pod "58298239-da8a-4443-a7a8-e3ceb9d7f3a8" (UID: "58298239-da8a-4443-a7a8-e3ceb9d7f3a8"). InnerVolumeSpecName "kube-api-access-q6tfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:48.180939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.180899    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6tfp\" (UniqueName: \"kubernetes.io/projected/58298239-da8a-4443-a7a8-e3ceb9d7f3a8-kube-api-access-q6tfp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:48.823234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.823159    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"]
Nov 03 19:49:48.833451 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.832994    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:48.836206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.836120    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"]
Nov 03 19:49:48.845595 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.845526    4531 generic.go:356] "Generic (PLEG): container finished" podID="58298239-da8a-4443-a7a8-e3ceb9d7f3a8" containerID="0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026" exitCode=137
Nov 03 19:49:48.848955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.845619    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-f2ca06"
Nov 03 19:49:48.848955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.845654    4531 scope.go:122] "RemoveContainer" containerID="0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026"
Nov 03 19:49:48.848955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.847223    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a536f35b\"/\"openshift-service-ca.crt\""
Nov 03 19:49:48.848955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.847412    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a536f35b\"/\"e2e-lifecycle-model-a536f35b-kserve-self-signed-certs\""
Nov 03 19:49:48.848955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.847421    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a536f35b\"/\"default-dockercfg-m75b7\""
Nov 03 19:49:48.848955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.847463    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a536f35b\"/\"kube-root-ca.crt\""
Nov 03 19:49:48.864101 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:49:48.864064    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-f2ca06\" is forbidden: User \"system:node:ip-10-0-131-234.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-234.us-east-2.compute.internal' and this object" podUID="58298239-da8a-4443-a7a8-e3ceb9d7f3a8" pod="opendatahub/test-curl-7096-f2ca06"
Nov 03 19:49:48.865150 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.864950    4531 scope.go:122] "RemoveContainer" containerID="0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026"
Nov 03 19:49:48.865455 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:49:48.865425    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026\": container with ID starting with 0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026 not found: ID does not exist" containerID="0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026"
Nov 03 19:49:48.865598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.865466    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026"} err="failed to get container status \"0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026\": rpc error: code = NotFound desc = could not find container \"0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026\": container with ID starting with 0ddcc49058a536a8021fff78bed4763130960bfbddf494912efefbc1acfdb026 not found: ID does not exist"
Nov 03 19:49:48.869218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.869167    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58298239-da8a-4443-a7a8-e3ceb9d7f3a8" path="/var/lib/kubelet/pods/58298239-da8a-4443-a7a8-e3ceb9d7f3a8/volumes"
Nov 03 19:49:48.886481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.886441    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:48.903855 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.903814    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-dshm\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:48.904116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.903952    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-home\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:48.904116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.904004    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-model-cache\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:48.904116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.904066    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzbm\" (UniqueName: \"kubernetes.io/projected/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-kube-api-access-wjzbm\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:48.904330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.904274    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tls-certs\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:48.904506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.904489    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tmp-dir\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:48.936953 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.936910    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7096-1f4900"]
Nov 03 19:49:48.945719 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:48.945682    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7096-1f4900"]
Nov 03 19:49:49.030198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.007275    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tmp-dir\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.030198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.007432    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-dshm\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.030198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.007574    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-home\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.030198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.007637    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-model-cache\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.030198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.007707    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzbm\" (UniqueName: \"kubernetes.io/projected/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-kube-api-access-wjzbm\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.030198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.007907    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tls-certs\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.099288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.078331    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tmp-dir\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.099288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.079411    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-model-cache\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.159196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.133307    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:49.199864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.194700    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-dshm\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.215217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.212446    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-home\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.238207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.216286    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tls-certs\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.301981 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.301602    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzbm\" (UniqueName: \"kubernetes.io/projected/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-kube-api-access-wjzbm\") pod \"e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") " pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.504202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.477363    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:49.903434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:49.903221    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:50.003127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:50.003078    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7096-1f4900" podUID="98e72f1a-7282-4d59-9749-ddd4b3f72c24" containerName="test-curl-7096-1f4900" containerID="cri-o://6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3" gracePeriod=2
Nov 03 19:49:50.006452 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:49:50.006395    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-1f4900\" is forbidden: User \"system:node:ip-10-0-131-234.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-234.us-east-2.compute.internal' and this object" podUID="98e72f1a-7282-4d59-9749-ddd4b3f72c24" pod="opendatahub/test-curl-7096-1f4900"
Nov 03 19:49:50.400579 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:50.400069    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:50.639193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:50.638957    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:50.714266 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:50.712405    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"]
Nov 03 19:49:50.730371 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:49:50.730334    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:49:50.951723 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:50.951691    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-95b4b7"
Nov 03 19:49:51.025127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.025089    4531 generic.go:356] "Generic (PLEG): container finished" podID="4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f" containerID="3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d" exitCode=137
Nov 03 19:49:51.025734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.025713    4531 scope.go:122] "RemoveContainer" containerID="3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d"
Nov 03 19:49:51.026007 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.025988    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-95b4b7"
Nov 03 19:49:51.032163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.032135    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" event={"ID":"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0","Type":"ContainerStarted","Data":"59fd59aa8f96ec15c4b7eee9d5cfd3eb3dbde5d8bdf436eff5a4c789b913db0a"}
Nov 03 19:49:51.032390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.032357    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" event={"ID":"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0","Type":"ContainerStarted","Data":"4ab56d188bb18d67096727aa8194c471df32fa8a73976444fba9688af3af22f0"}
Nov 03 19:49:51.051880 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.051850    4531 scope.go:122] "RemoveContainer" containerID="3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d"
Nov 03 19:49:51.052436 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:49:51.052410    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d\": container with ID starting with 3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d not found: ID does not exist" containerID="3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d"
Nov 03 19:49:51.052580 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.052553    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d"} err="failed to get container status \"3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d\": rpc error: code = NotFound desc = could not find container \"3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d\": container with ID starting with 3dcad422f45da5ab32f9982036ec7f4d31e3810dab5778f0bf245a410893f96d not found: ID does not exist"
Nov 03 19:49:51.060286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.060252    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f-kube-api-access-qsjkt\" (UniqueName: \"kubernetes.io/projected/4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f-kube-api-access-qsjkt\") pod \"4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f\" (UID: \"4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f\") "
Nov 03 19:49:51.068681 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.068616    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f-kube-api-access-qsjkt" pod "4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f" (UID: "4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f"). InnerVolumeSpecName "kube-api-access-qsjkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:51.079054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.078992    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" podStartSLOduration=3.078970767 podStartE2EDuration="3.078970767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:51.06329315 +0000 UTC m=+1104.333416475" watchObservedRunningTime="2026-11-03 19:49:51.078970767 +0000 UTC m=+1104.349094064"
Nov 03 19:49:51.154243 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.154066    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:51.163661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.163624    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsjkt\" (UniqueName: \"kubernetes.io/projected/4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f-kube-api-access-qsjkt\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:51.328221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.321473    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:51.439029 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:49:51.438986    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-95b4b7\" is forbidden: User \"system:node:ip-10-0-131-234.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-234.us-east-2.compute.internal' and this object" podUID="4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f" pod="opendatahub/test-curl-7096-95b4b7"
Nov 03 19:49:51.456475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.456442    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:51.479597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:51.479535    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:52.735201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:52.735010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:52.888920 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:52.888336    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-1f4900"
Nov 03 19:49:52.890834 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:49:52.890783    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7096-95b4b7\" is forbidden: User \"system:node:ip-10-0-131-234.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-234.us-east-2.compute.internal' and this object" podUID="4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f" pod="opendatahub/test-curl-7096-95b4b7"
Nov 03 19:49:52.980270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:52.977860    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e72f1a-7282-4d59-9749-ddd4b3f72c24-kube-api-access-swvkp\" (UniqueName: \"kubernetes.io/projected/98e72f1a-7282-4d59-9749-ddd4b3f72c24-kube-api-access-swvkp\") pod \"98e72f1a-7282-4d59-9749-ddd4b3f72c24\" (UID: \"98e72f1a-7282-4d59-9749-ddd4b3f72c24\") "
Nov 03 19:49:52.980766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:52.980734    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f" path="/var/lib/kubelet/pods/4cc1c5d9-d2e1-40a6-b6fc-5ad4b127a85f/volumes"
Nov 03 19:49:52.993001 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:52.992972    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e72f1a-7282-4d59-9749-ddd4b3f72c24-kube-api-access-swvkp" pod "98e72f1a-7282-4d59-9749-ddd4b3f72c24" (UID: "98e72f1a-7282-4d59-9749-ddd4b3f72c24"). InnerVolumeSpecName "kube-api-access-swvkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:53.082199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.080349    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:53.082381 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.082266    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvkp\" (UniqueName: \"kubernetes.io/projected/98e72f1a-7282-4d59-9749-ddd4b3f72c24-kube-api-access-swvkp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:53.127195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.126864    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7n48w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:49:53.127195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.126942    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" podUID="bd352072-e76d-4d10-93a1-a71b2e0495ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:53.199385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.199344    4531 generic.go:356] "Generic (PLEG): container finished" podID="98e72f1a-7282-4d59-9749-ddd4b3f72c24" containerID="6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3" exitCode=137
Nov 03 19:49:53.200887 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.200861    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7096-1f4900"
Nov 03 19:49:53.214452 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.214424    4531 scope.go:122] "RemoveContainer" containerID="6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3"
Nov 03 19:49:53.635213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.606920    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:53.849426 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.849379    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:49:53.890204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.888628    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:53.910303 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.910256    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:53.918287 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.918245    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:53.957774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:53.949691    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:54.038586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:54.038555    4531 scope.go:122] "RemoveContainer" containerID="6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3"
Nov 03 19:49:54.057233 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:49:54.047550    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3\": container with ID starting with 6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3 not found: ID does not exist" containerID="6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3"
Nov 03 19:49:54.057233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:54.047596    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3"} err="failed to get container status \"6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3\": rpc error: code = NotFound desc = could not find container \"6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3\": container with ID starting with 6d653e319a0e243e529078a5beef82d4e6d9a128b3ea0d4d344655b0565d1ad3 not found: ID does not exist"
Nov 03 19:49:54.104238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:54.083593    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:54.104238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:54.084273    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:49:54.180507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:54.180318    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:54.281198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:54.278089    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:54.388028 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:54.387996    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:49:54.486194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:54.484285    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:54.861236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:54.861151    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e72f1a-7282-4d59-9749-ddd4b3f72c24" path="/var/lib/kubelet/pods/98e72f1a-7282-4d59-9749-ddd4b3f72c24/volumes"
Nov 03 19:49:55.584462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:55.584431    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:56.858314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:56.858191    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:57.781331 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:57.781292    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:58.637544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:58.637421    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:59.024015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:59.023933    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"]
Nov 03 19:49:59.029633 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:59.029594    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6158f102-r6mqq"]
Nov 03 19:49:59.285227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:59.285059    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:59.435736 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:59.435700    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:59.485557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:59.484313    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:59.566431 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:59.566358    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:49:59.989436 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:49:59.961808    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:00.359259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:00.349063    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:00.440894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:00.440860    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:50:00.650993 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:00.650928    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:00.861233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:00.861193    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009fab84-7b8a-4199-9c69-b99aa4e41eeb" path="/var/lib/kubelet/pods/009fab84-7b8a-4199-9c69-b99aa4e41eeb/volumes"
Nov 03 19:50:01.669202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:01.660480    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:02.889223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:02.888748    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:03.381957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:03.381927    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:03.510113 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:03.510061    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:04.059196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.053119    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:04.689065 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.689029    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"]
Nov 03 19:50:04.689906 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.689889    4531 cpu_manager.go:414] "containerMap: removing container" podUID="98e72f1a-7282-4d59-9749-ddd4b3f72c24" containerName="test-curl-7096-1f4900"
Nov 03 19:50:04.689962 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.689908    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e72f1a-7282-4d59-9749-ddd4b3f72c24" containerName="test-curl-7096-1f4900"
Nov 03 19:50:04.690097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.690086    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e72f1a-7282-4d59-9749-ddd4b3f72c24" containerName="test-curl-7096-1f4900"
Nov 03 19:50:04.698043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.698022    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:50:04.706335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.705827    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3c7998\""
Nov 03 19:50:04.710444 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.710410    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"]
Nov 03 19:50:04.737668 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.737625    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:04.751237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.751201    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"]
Nov 03 19:50:04.756885 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.756855    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:50:04.773430 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.771430    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3c7998-dockercfg-zsdpv\""
Nov 03 19:50:04.781222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.779843    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"]
Nov 03 19:50:04.800241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.800206    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq\" (UID: \"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:50:04.800413 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.800393    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvlv\" (UniqueName: \"kubernetes.io/projected/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-kube-api-access-vlvlv\") pod \"payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq\" (UID: \"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:50:04.902164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.902123    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq\" (UID: \"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:50:04.902372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.902245    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x86nl\" (UniqueName: \"kubernetes.io/projected/961783a9-751a-4630-b785-e43e96ce84b3-kube-api-access-x86nl\") pod \"payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m\" (UID: \"961783a9-751a-4630-b785-e43e96ce84b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:50:04.902372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.902307    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961783a9-751a-4630-b785-e43e96ce84b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m\" (UID: \"961783a9-751a-4630-b785-e43e96ce84b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:50:04.902600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.902565    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvlv\" (UniqueName: \"kubernetes.io/projected/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-kube-api-access-vlvlv\") pod \"payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq\" (UID: \"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:50:04.902998 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.902979    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq\" (UID: \"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:50:04.907364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.907328    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:04.910020 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:04.909995    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvlv\" (UniqueName: \"kubernetes.io/projected/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-kube-api-access-vlvlv\") pod \"payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq\" (UID: \"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:50:05.021258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:05.015042    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x86nl\" (UniqueName: \"kubernetes.io/projected/961783a9-751a-4630-b785-e43e96ce84b3-kube-api-access-x86nl\") pod \"payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m\" (UID: \"961783a9-751a-4630-b785-e43e96ce84b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:50:05.021258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:05.015132    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961783a9-751a-4630-b785-e43e96ce84b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m\" (UID: \"961783a9-751a-4630-b785-e43e96ce84b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:50:05.045202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:05.036948    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:50:05.045202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:05.037491    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961783a9-751a-4630-b785-e43e96ce84b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m\" (UID: \"961783a9-751a-4630-b785-e43e96ce84b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:50:05.054782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:05.051226    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x86nl\" (UniqueName: \"kubernetes.io/projected/961783a9-751a-4630-b785-e43e96ce84b3-kube-api-access-x86nl\") pod \"payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m\" (UID: \"961783a9-751a-4630-b785-e43e96ce84b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:50:05.108195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:05.100265    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:50:05.125198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:05.118311    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:05.646841 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:50:05.646808    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cbbfb1_9923_4f2d_96d9_5a5ae9d3231f.slice/crio-ef63c30da5dafbf27c31fad051c7a6be3b13d8bb60edd5a95f127bb1555784d9 WatchSource:0}: Error finding container ef63c30da5dafbf27c31fad051c7a6be3b13d8bb60edd5a95f127bb1555784d9: Status 404 returned error can't find the container with id ef63c30da5dafbf27c31fad051c7a6be3b13d8bb60edd5a95f127bb1555784d9
Nov 03 19:50:05.647456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:05.647383    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"]
Nov 03 19:50:05.718740 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:50:05.718715    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961783a9_751a_4630_b785_e43e96ce84b3.slice/crio-f494bc73548142a96ec20782a09dddd2451c3eef57b59b18da8450500a4f2d67 WatchSource:0}: Error finding container f494bc73548142a96ec20782a09dddd2451c3eef57b59b18da8450500a4f2d67: Status 404 returned error can't find the container with id f494bc73548142a96ec20782a09dddd2451c3eef57b59b18da8450500a4f2d67
Nov 03 19:50:05.721027 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:05.721007    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"]
Nov 03 19:50:05.811327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:05.803659    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:06.024712 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.024678    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"]
Nov 03 19:50:06.030271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.030245    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c764a998-c4rfm"]
Nov 03 19:50:06.463196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.457732    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:06.545365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.544909    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq" event={"ID":"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f","Type":"ContainerStarted","Data":"ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be"}
Nov 03 19:50:06.545365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.545014    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq" event={"ID":"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f","Type":"ContainerStarted","Data":"ef63c30da5dafbf27c31fad051c7a6be3b13d8bb60edd5a95f127bb1555784d9"}
Nov 03 19:50:06.545850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.545826    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:50:06.550221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.548361    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m" event={"ID":"961783a9-751a-4630-b785-e43e96ce84b3","Type":"ContainerStarted","Data":"e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0"}
Nov 03 19:50:06.550221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.548412    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m" event={"ID":"961783a9-751a-4630-b785-e43e96ce84b3","Type":"ContainerStarted","Data":"f494bc73548142a96ec20782a09dddd2451c3eef57b59b18da8450500a4f2d67"}
Nov 03 19:50:06.550221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.548614    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:50:06.560299 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.560254    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq" podStartSLOduration=2.56024238 podStartE2EDuration="2.56024238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:06.560103863 +0000 UTC m=+1119.830227157" watchObservedRunningTime="2026-11-03 19:50:06.56024238 +0000 UTC m=+1119.830365674"
Nov 03 19:50:06.576502 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.576444    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m" podStartSLOduration=2.576426053 podStartE2EDuration="2.576426053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:06.574314168 +0000 UTC m=+1119.844437459" watchObservedRunningTime="2026-11-03 19:50:06.576426053 +0000 UTC m=+1119.846549349"
Nov 03 19:50:06.861845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:06.861759    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc192a79-2ff6-43cb-a4a4-9ecf03c76d02" path="/var/lib/kubelet/pods/bc192a79-2ff6-43cb-a4a4-9ecf03c76d02/volumes"
Nov 03 19:50:07.258750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:07.258712    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:08.088529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:08.088489    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:08.268383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:08.268351    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:08.939199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:08.919659    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:09.715259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:09.706564    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:10.795400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:10.795353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:11.436144 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:11.435067    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:12.339450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:12.339368    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:13.180111 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:13.180074    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:13.963946 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:13.963908    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:14.053973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:14.053884    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"]
Nov 03 19:50:14.063700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:14.063666    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eef7ec0-5cqw8"]
Nov 03 19:50:14.126789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:14.126752    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:14.398413 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:14.398302    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:14.916827 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:14.916790    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716a697b-354d-4e08-baeb-b4a3aed59613" path="/var/lib/kubelet/pods/716a697b-354d-4e08-baeb-b4a3aed59613/volumes"
Nov 03 19:50:14.958352 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:14.958322    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:15.115198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:15.112975    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:15.784039 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:15.783993    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:15.953306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:15.951685    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:16.562925 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:16.562887    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:17.180197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:17.171854    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:17.562466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:17.562384    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:50:17.562736 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:17.562709    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:50:17.914637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:17.914601    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:18.565218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:18.565160    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:19.244055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:19.243986    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:19.826785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:19.826736    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:20.459448 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:20.459415    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:20.622455 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:50:20.622419    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:50:21.190199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:21.169490    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:21.915240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:21.915156    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:22.623196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:22.619623    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:23.070164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.070130    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wfnhm-m7rk8"]
Nov 03 19:50:23.075486 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.075459    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.078633 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.078611    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wrft7\""
Nov 03 19:50:23.080579 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.080555    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wfnhm-m7rk8"]
Nov 03 19:50:23.087627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.085936    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 19:50:23.114951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.114915    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 19:50:23.214033 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.213996    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-service-ca-bundle\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.214268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.214150    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/64f7cc83-420d-4739-ace1-db3694d9dca1-archives-path\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.214328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.214261    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kdd6\" (UniqueName: \"kubernetes.io/projected/64f7cc83-420d-4739-ace1-db3694d9dca1-kube-api-access-5kdd6\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.214540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.214521    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-trusted-ca-bundle\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.247859 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.247817    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:23.317195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.316893    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-service-ca-bundle\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.317195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.317046    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/64f7cc83-420d-4739-ace1-db3694d9dca1-archives-path\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.317496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.317390    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdd6\" (UniqueName: \"kubernetes.io/projected/64f7cc83-420d-4739-ace1-db3694d9dca1-kube-api-access-5kdd6\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.317706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.317681    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-trusted-ca-bundle\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.320531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.320462    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/64f7cc83-420d-4739-ace1-db3694d9dca1-archives-path\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.321337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.321282    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-service-ca-bundle\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.324709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.322007    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-trusted-ca-bundle\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.344984 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.344797    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kdd6\" (UniqueName: \"kubernetes.io/projected/64f7cc83-420d-4739-ace1-db3694d9dca1-kube-api-access-5kdd6\") pod \"periodic-gathering-wfnhm-m7rk8\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") " pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.405813 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.405777    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:50:23.697571 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:50:23.697537    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f7cc83_420d_4739_ace1_db3694d9dca1.slice/crio-4840b51a50ba82172249ed358cdfee5dea16e8ba13a2034b4aef0d63d8817ed6 WatchSource:0}: Error finding container 4840b51a50ba82172249ed358cdfee5dea16e8ba13a2034b4aef0d63d8817ed6: Status 404 returned error can't find the container with id 4840b51a50ba82172249ed358cdfee5dea16e8ba13a2034b4aef0d63d8817ed6
Nov 03 19:50:23.698891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.698865    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wfnhm-m7rk8"]
Nov 03 19:50:23.842914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.842881    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wfnhm-m7rk8" event={"ID":"64f7cc83-420d-4739-ace1-db3694d9dca1","Type":"ContainerStarted","Data":"e32c951cc27fb39f598e0a20a20545fcab71436e869dbcc33d28746a18f44944"}
Nov 03 19:50:23.843083 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.842935    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wfnhm-m7rk8" event={"ID":"64f7cc83-420d-4739-ace1-db3694d9dca1","Type":"ContainerStarted","Data":"4840b51a50ba82172249ed358cdfee5dea16e8ba13a2034b4aef0d63d8817ed6"}
Nov 03 19:50:23.889522 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:23.888992    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:24.901228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:24.896603    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:25.628140 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:25.626358    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:26.307567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:26.307535    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:26.928344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:26.928307    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:27.602206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:27.601055    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:28.377197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:28.377139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:28.540399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:28.540367    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:29.169186 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:29.169149    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:29.319466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:29.319430    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:29.497986 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:29.497873    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wfnhm-m7rk8" podStartSLOduration=6.497852599 podStartE2EDuration="6.497852599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:23.901667386 +0000 UTC m=+1137.171790682" watchObservedRunningTime="2026-11-03 19:50:29.497852599 +0000 UTC m=+1142.767975897"
Nov 03 19:50:29.620421 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:29.614984    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:29.761260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:29.761156    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"]
Nov 03 19:50:30.189197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.161299    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.260197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.257722    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-82b108-openshift-default-dockercfg-2s4w7\""
Nov 03 19:50:30.305486 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.305445    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"]
Nov 03 19:50:30.479255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.462417    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"]
Nov 03 19:50:30.479520 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.462565    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:30.481060 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.462646    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:30.516658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.485337    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.160:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:30.516658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.491405    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r62qc\" (UniqueName: \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-kube-api-access-r62qc\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.516658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.491548    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-envoy\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.516658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.491586    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-certs\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.516658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.491746    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-data\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.516658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.491862    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-token\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.516658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.491913    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ce009d0-6fa8-4449-8517-42a82817a60f-istiod-ca-cert\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.516658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.491990    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-socket\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.516658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.492026    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-credential-socket\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.516658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.492056    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-podinfo\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.576540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.576361    4531 scope.go:122] "RemoveContainer" containerID="de44e64ee812fb7f859de31f4d6e38c244d82adb2c07b8cc9cdd70fc903b9209"
Nov 03 19:50:30.665834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.658713    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:30.675938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.675897    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r62qc\" (UniqueName: \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-kube-api-access-r62qc\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.676600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.676569    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-envoy\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.677221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.677197    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-certs\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.678964 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.678939    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-data\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.694015    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-token\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.694108    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ce009d0-6fa8-4449-8517-42a82817a60f-istiod-ca-cert\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.694232    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-socket\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.694272    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-credential-socket\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.694307    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-podinfo\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.697490    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-podinfo\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.681234    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-certs\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.686393    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-data\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.698713    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ce009d0-6fa8-4449-8517-42a82817a60f-istiod-ca-cert\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.693710    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r62qc\" (UniqueName: \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-kube-api-access-r62qc\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.699059    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-socket\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.699406    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-credential-socket\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.708198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.705130    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-token\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.814812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.814782    4531 scope.go:122] "RemoveContainer" containerID="bfd2cb980961a49b8910914b5a5997c86a33eb7bcc8c04f0432cd5c9e30eaab5"
Nov 03 19:50:30.833184 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.833140    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3013387d-fqvsk"]
Nov 03 19:50:30.856069 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.856028    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-envoy\") pod \"e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") " pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:30.868024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.867997    4531 scope.go:122] "RemoveContainer" containerID="1bce39b48744e3ed182df69b30f8253a5b6e0fa09260de9110d60b7128200384"
Nov 03 19:50:30.879487 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.879450    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a877371b-020a-44fd-b07b-80bd5a58c75a" path="/var/lib/kubelet/pods/a877371b-020a-44fd-b07b-80bd5a58c75a/volumes"
Nov 03 19:50:30.904939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.904907    4531 scope.go:122] "RemoveContainer" containerID="0bc9910f35d4fdb665c69566014af6e49c194c6dada9bb8f64cd4fae48125b67"
Nov 03 19:50:30.926412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.926388    4531 scope.go:122] "RemoveContainer" containerID="e58b3b5069a1632d5722bad0d501dbbdea75f86049f8fbe76827b57178ad7a86"
Nov 03 19:50:30.937258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.937202    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:30.960396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:30.960359    4531 scope.go:122] "RemoveContainer" containerID="6a073ece546657652cc390019c8cfbd9225b74ba97a25b76ecc2b1641dabd35a"
Nov 03 19:50:31.097262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.092917    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:31.109502 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.107946    4531 scope.go:122] "RemoveContainer" containerID="0aead45c81b634d99e1117fdca48ab157e607779c485d2cfd9adf79eb640dfa2"
Nov 03 19:50:31.128800 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.128425    4531 scope.go:122] "RemoveContainer" containerID="afc45e5c9d89a7b2135ecdd3de5db58723d08876f06f4b1ecb0bb6884f934072"
Nov 03 19:50:31.128800 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.128575    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:31.160636 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.160464    4531 scope.go:122] "RemoveContainer" containerID="c82952a670dc73aeb6369b39b48fef67acdece780210900ebf56c90b7481efb2"
Nov 03 19:50:31.196153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.195770    4531 scope.go:122] "RemoveContainer" containerID="2508335227f02d74ac65733875705690595d12075e6b426f3b7c7efbd85b9699"
Nov 03 19:50:31.218011 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.217971    4531 scope.go:122] "RemoveContainer" containerID="72203e10b28b33de423f8682a232e37278788b4823932c404873bf4b4758c633"
Nov 03 19:50:31.245490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.245425    4531 scope.go:122] "RemoveContainer" containerID="1eb7dcebae53c86aea867dc7969d158dbb2affea88b960e70cd4f33e74178d8b"
Nov 03 19:50:31.274569 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.274535    4531 scope.go:122] "RemoveContainer" containerID="9d7895667addc8ef631538583dd021aa5649601439da0e557af00a98dd906209"
Nov 03 19:50:31.314669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.314635    4531 scope.go:122] "RemoveContainer" containerID="db6fd52270f8a92401cea88a52d19623315e6c7f6f3a9915752e96171901f714"
Nov 03 19:50:31.476397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.476375    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"]
Nov 03 19:50:31.489476 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:50:31.489438    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce009d0_6fa8_4449_8517_42a82817a60f.slice/crio-6f26b7d40a635fd0a051bd7dd9e5eedba27176135e4b5519ece2d8b59b02d781 WatchSource:0}: Error finding container 6f26b7d40a635fd0a051bd7dd9e5eedba27176135e4b5519ece2d8b59b02d781: Status 404 returned error can't find the container with id 6f26b7d40a635fd0a051bd7dd9e5eedba27176135e4b5519ece2d8b59b02d781
Nov 03 19:50:31.499958 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.499908    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:31.500076 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.500017    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:31.500076 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.500065    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:31.574598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:31.574557    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" event={"ID":"6ce009d0-6fa8-4449-8517-42a82817a60f","Type":"ContainerStarted","Data":"6f26b7d40a635fd0a051bd7dd9e5eedba27176135e4b5519ece2d8b59b02d781"}
Nov 03 19:50:32.069198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:32.057585    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:32.700750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:32.700704    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" event={"ID":"6ce009d0-6fa8-4449-8517-42a82817a60f","Type":"ContainerStarted","Data":"5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d"}
Nov 03 19:50:32.782198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:32.760087    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" podStartSLOduration=3.760062664 podStartE2EDuration="3.760062664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:32.722831621 +0000 UTC m=+1145.992954918" watchObservedRunningTime="2026-11-03 19:50:32.760062664 +0000 UTC m=+1146.030185959"
Nov 03 19:50:33.097246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.093740    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"]
Nov 03 19:50:33.142209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.132392    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:33.164197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.147466    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:33.179202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.176971    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24844a64-5ftq8"]
Nov 03 19:50:33.546198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.522416    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:50:33.546198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.522497    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl" podUID="e5b033e2-ebcc-4112-99d6-d9398e5ac835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:33.546198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.538307    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-g74zd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:50:33.546198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.538365    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:33.653223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.644542    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:33.772199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.758974    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:33.776207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.773188    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:50:33.952203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.938283    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:34.035199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:33.997365    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:50:34.035199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:34.010535    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:34.035199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:34.015058    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:50:34.035199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:34.015382    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:50:34.194200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:34.188303    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:34.401204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:34.400747    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:50:34.473199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:34.469398    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:34.497471 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:34.496850    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.160:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:34.661930 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:34.661825    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:34.861892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:34.861860    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e92b7d4-cc64-44ea-99b3-1b3db7473f7c" path="/var/lib/kubelet/pods/5e92b7d4-cc64-44ea-99b3-1b3db7473f7c/volumes"
Nov 03 19:50:35.628612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:35.628558    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:36.390820 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.385836    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:36.521081 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.521046    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:36.714030 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.713944    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"]
Nov 03 19:50:36.716488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.716457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:36.742765 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.742722    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"]
Nov 03 19:50:36.742957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.742834    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:36.748340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.748311    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-fa9526-openshift-default-dockercfg-gbpc5\""
Nov 03 19:50:36.879200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.868572    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-podinfo\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:36.879662 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.879600    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-data\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:36.879884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.879860    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-certs\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:36.880088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.880063    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-credential-socket\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:36.897394 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.897352    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-envoy\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:36.897743 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.897721    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-socket\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:36.897912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.897893    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9597b9a-67e3-492c-afb7-dc2e544b156d-istiod-ca-cert\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:36.898086 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.898058    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-token\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:36.898500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.898469    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqrrt\" (UniqueName: \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-kube-api-access-tqrrt\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:36.911890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:36.911843    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:37.018692 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.018595    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-podinfo\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.018943 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.018919    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-data\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.019065 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.019050    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-certs\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.019227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.019211    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-credential-socket\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.019428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.019412    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-envoy\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.019618 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.019603    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-socket\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.019742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.019728    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9597b9a-67e3-492c-afb7-dc2e544b156d-istiod-ca-cert\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.019849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.019835    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-token\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.020066 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.020050    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrrt\" (UniqueName: \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-kube-api-access-tqrrt\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.028210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.021600    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-credential-socket\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.030015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.029986    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-data\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.030457 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.030433    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-certs\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.073193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.031113    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9597b9a-67e3-492c-afb7-dc2e544b156d-istiod-ca-cert\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.073193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.050390    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-socket\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.074966 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.074733    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:37.093319 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.082465    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-envoy\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.186039 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.185969    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-podinfo\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.222573 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.221447    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrrt\" (UniqueName: \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-kube-api-access-tqrrt\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.242322 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.242235    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-token\") pod \"e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") " pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.385247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.378827    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:37.663413 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.651778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:38.002193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:37.984849    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:38.246213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:38.246093    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:38.317785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:38.317727    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:38.318071 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:38.318038    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:38.318253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:38.318218    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:38.322318 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:38.321641    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"]
Nov 03 19:50:38.550286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:38.550192    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:39.015907 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:39.015863    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" event={"ID":"b9597b9a-67e3-492c-afb7-dc2e544b156d","Type":"ContainerStarted","Data":"246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660"}
Nov 03 19:50:39.016344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:39.015929    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" event={"ID":"b9597b9a-67e3-492c-afb7-dc2e544b156d","Type":"ContainerStarted","Data":"13b55486520515aa7727c1bfdc58539f52257cbf692ad4dc05328ac92cc18471"}
Nov 03 19:50:39.035765 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:39.035713    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" podStartSLOduration=3.035696161 podStartE2EDuration="3.035696161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:39.032622137 +0000 UTC m=+1152.302745434" watchObservedRunningTime="2026-11-03 19:50:39.035696161 +0000 UTC m=+1152.305819458"
Nov 03 19:50:39.379677 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:39.379649    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:39.385759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:39.385734    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:39.681258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:39.675868    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:39.940610 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:39.940525    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:40.024829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:40.024797    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:40.025772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:40.025746    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:50:40.079101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:40.079039    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:40.513196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:40.509004    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:41.253783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:41.253737    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:41.414193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:41.413999    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:41.590251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:41.581820    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:41.770195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:41.767727    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:41.925915 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:41.925877    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:42.108847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:42.107961    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:42.263470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:42.257008    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:42.854982 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:42.854948    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:43.015197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:43.014682    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:43.443243 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:43.440851    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:43.959447 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:43.959416    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"]
Nov 03 19:50:43.960272 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:43.960246    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:43.966843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:43.966818    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:50:43.969678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:43.969645    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"]
Nov 03 19:50:43.970296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:43.970271    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2b73c810-dockercfg-rgfw4\""
Nov 03 19:50:44.049194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:44.048460    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2b73c810\""
Nov 03 19:50:44.223616 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:44.222014    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:44.225024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:44.224995    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f432b8b2-d219-4deb-ba65-4604103a176e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg\" (UID: \"f432b8b2-d219-4deb-ba65-4604103a176e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:50:44.225841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:44.225816    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65g2f\" (UniqueName: \"kubernetes.io/projected/f432b8b2-d219-4deb-ba65-4604103a176e-kube-api-access-65g2f\") pod \"payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg\" (UID: \"f432b8b2-d219-4deb-ba65-4604103a176e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:50:44.328043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:44.327999    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f432b8b2-d219-4deb-ba65-4604103a176e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg\" (UID: \"f432b8b2-d219-4deb-ba65-4604103a176e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:50:44.328799 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:44.328768    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65g2f\" (UniqueName: \"kubernetes.io/projected/f432b8b2-d219-4deb-ba65-4604103a176e-kube-api-access-65g2f\") pod \"payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg\" (UID: \"f432b8b2-d219-4deb-ba65-4604103a176e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:50:44.330529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:44.330472    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f432b8b2-d219-4deb-ba65-4604103a176e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg\" (UID: \"f432b8b2-d219-4deb-ba65-4604103a176e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:50:44.383207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:44.375325    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:44.383207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:44.377568    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65g2f\" (UniqueName: \"kubernetes.io/projected/f432b8b2-d219-4deb-ba65-4604103a176e-kube-api-access-65g2f\") pod \"payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg\" (UID: \"f432b8b2-d219-4deb-ba65-4604103a176e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:50:44.599024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:44.589955    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:50:45.042652 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:45.042597    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:45.168931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:45.167454    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"]
Nov 03 19:50:45.303785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:45.303693    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg" event={"ID":"f432b8b2-d219-4deb-ba65-4604103a176e","Type":"ContainerStarted","Data":"ef2b5baf393686257d26f9f9de33397341eed8597768d8ac98f57c47344f9467"}
Nov 03 19:50:45.856761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:45.856729    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:46.316053 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:46.316015    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg" event={"ID":"f432b8b2-d219-4deb-ba65-4604103a176e","Type":"ContainerStarted","Data":"3c62da78e9bc3e5f41277f516a8ddcadfffbccfb79573bc22341442ff15d9b9c"}
Nov 03 19:50:46.316269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:46.316084    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:50:46.335416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:46.335368    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg" podStartSLOduration=3.3353557670000002 podStartE2EDuration="3.335355767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:46.331460501 +0000 UTC m=+1159.601583796" watchObservedRunningTime="2026-11-03 19:50:46.335355767 +0000 UTC m=+1159.605479061"
Nov 03 19:50:46.731112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:46.731072    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:47.500223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:47.473839    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:48.287438 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:48.287388    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:49.028146 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:49.028091    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"]
Nov 03 19:50:49.060201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:49.056224    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:49.095872 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:49.095828    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268e9995-sgxjv"]
Nov 03 19:50:50.002915 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:50.002877    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:50.578322 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:50:50.578285    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:50:50.668252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:50.661062    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:50.862764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:50.862677    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="970b169f-9ebf-4fed-b9d1-44ee2f761c58" path="/var/lib/kubelet/pods/970b169f-9ebf-4fed-b9d1-44ee2f761c58/volumes"
Nov 03 19:50:51.361542 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:51.355721    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:52.022575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:52.022513    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:52.226379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:52.226343    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:52.351297 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:52.351224    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:53.059459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:53.059420    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:54.232608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:54.229093    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:54.375812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:54.375762    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:55.156962 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:55.156929    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:56.066372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.066340    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"]
Nov 03 19:50:56.324354 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.316224    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:56.373147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.373113    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"]
Nov 03 19:50:56.373431 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.373288    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.376341 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.376314    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2b73c810-serving-cert\""
Nov 03 19:50:56.444960 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.444923    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26a827ee-7cc3-4a48-947f-10b965165761-openshift-service-ca\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.445139 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.445037    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jmsx\" (UniqueName: \"kubernetes.io/projected/26a827ee-7cc3-4a48-947f-10b965165761-kube-api-access-9jmsx\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.445244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.445219    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-metrics-certs\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.445358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.445342    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-maas-api-tls\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.453348 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.453311    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:56.547979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.547941    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26a827ee-7cc3-4a48-947f-10b965165761-openshift-service-ca\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.548147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.547998    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jmsx\" (UniqueName: \"kubernetes.io/projected/26a827ee-7cc3-4a48-947f-10b965165761-kube-api-access-9jmsx\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.548147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.548117    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-metrics-certs\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.548332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.548269    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-maas-api-tls\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.548736 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.548710    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26a827ee-7cc3-4a48-947f-10b965165761-openshift-service-ca\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.550716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.550693    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-metrics-certs\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.550821 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.550719    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-maas-api-tls\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.555687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.555662    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jmsx\" (UniqueName: \"kubernetes.io/projected/26a827ee-7cc3-4a48-947f-10b965165761-kube-api-access-9jmsx\") pod \"maas-api-e2e-mt-2b73c810-6db878bc98-xqk22\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:56.583633 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.583548    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:56.694040 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:56.694006    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:57.253252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:57.253166    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:57.328597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:57.328568    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:50:57.700210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:57.676482    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"]
Nov 03 19:50:57.734105 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:50:57.733972    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a827ee_7cc3_4a48_947f_10b965165761.slice/crio-4441b7555f3e3f618a0f72a8b64f81676677f5911fb940c2f34ef3fc7d5133b9 WatchSource:0}: Error finding container 4441b7555f3e3f618a0f72a8b64f81676677f5911fb940c2f34ef3fc7d5133b9: Status 404 returned error can't find the container with id 4441b7555f3e3f618a0f72a8b64f81676677f5911fb940c2f34ef3fc7d5133b9
Nov 03 19:50:57.862000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:57.861959    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22" event={"ID":"26a827ee-7cc3-4a48-947f-10b965165761","Type":"ContainerStarted","Data":"4441b7555f3e3f618a0f72a8b64f81676677f5911fb940c2f34ef3fc7d5133b9"}
Nov 03 19:50:58.208201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:58.200610    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:58.877839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:58.877794    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22" event={"ID":"26a827ee-7cc3-4a48-947f-10b965165761","Type":"ContainerStarted","Data":"84e91ca4941c69f08c0ac98d92e44b96b9518b8a32a0083aee8077233a060168"}
Nov 03 19:50:58.886197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:58.879318    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:50:58.913285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:58.910275    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22" podStartSLOduration=2.452682911 podStartE2EDuration="2.910239385s" totalImagesPullingTime="457.556474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:58.898928484 +0000 UTC m=+1172.169051780" watchObservedRunningTime="2026-11-03 19:50:58.910239385 +0000 UTC m=+1172.180362681"
Nov 03 19:50:58.941951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:58.941918    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:59.770198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:50:59.766407    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:00.466498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:00.466457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:01.222423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:01.222381    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:01.860728 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:01.860688    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:02.614975 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:02.614061    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:03.246196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:03.245690    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:03.906467 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:03.906424    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:04.545759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:04.545706    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:05.238872 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:05.238763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:05.852843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:05.852804    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:06.004192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:06.004151    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:51:06.028375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:06.028337    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"]
Nov 03 19:51:06.035407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:06.035378    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"]
Nov 03 19:51:06.048873 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:06.048843    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3e60c3-qw4f6"]
Nov 03 19:51:06.057817 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:06.057782    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e116f95-wpnwg"]
Nov 03 19:51:06.829458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:06.829415    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:06.968297 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:06.966331    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd16332-cc45-4642-8761-d5481a9a7453" path="/var/lib/kubelet/pods/5fd16332-cc45-4642-8761-d5481a9a7453/volumes"
Nov 03 19:51:06.969754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:06.969725    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e1961a-46b9-4994-9887-80b5236bfbe2" path="/var/lib/kubelet/pods/b5e1961a-46b9-4994-9887-80b5236bfbe2/volumes"
Nov 03 19:51:07.815335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:07.815299    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:08.910959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:08.907285    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:09.793791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:09.788768    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:10.288890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:10.288853    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:10.821865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:10.814772    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:11.474335 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:11.474272    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:12.535912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.535876    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"]
Nov 03 19:51:12.556473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.556422    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.559682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.559651    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-10572d-openshift-default-dockercfg-d5wdl\""
Nov 03 19:51:12.561880 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.561849    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"]
Nov 03 19:51:12.679641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.678958    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-socket\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.679641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.679357    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-credential-socket\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.792233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.792123    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-certs\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.802199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.792662    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-credential-socket\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.822726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.804751    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-credential-socket\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.823185 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.823027    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-data\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.823453 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.823432    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-envoy\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.838287 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.823764    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfx57\" (UniqueName: \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-kube-api-access-dfx57\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.849547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.849515    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-socket\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.850272 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.850144    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-socket\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.872715 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.872681    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-token\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.873077 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.873055    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-podinfo\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.873321 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.873306    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istiod-ca-cert\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.986193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.975789    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-podinfo\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.986193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.983815    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istiod-ca-cert\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.986193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.984040    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-certs\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.986193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.984295    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-data\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.986193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.984342    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-envoy\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.986193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.984548    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfx57\" (UniqueName: \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-kube-api-access-dfx57\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.986193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.984646    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-token\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:12.999445 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.999406    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-data\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:13.000054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:12.999983    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-certs\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:13.003510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:13.003485    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-token\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:13.023364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:13.023328    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istiod-ca-cert\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:13.023628 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:13.023367    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-envoy\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:13.024439 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:13.024363    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-podinfo\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:13.043262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:13.035774    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfx57\" (UniqueName: \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-kube-api-access-dfx57\") pod \"e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") " pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:13.180570 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:13.180526    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:13.581072 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:51:13.581030    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc9c2e9_6cdc_435d_8ce8_4a1fae9e0bce.slice/crio-324aab5698f88b3132539aa1b04d8abb77dd19d1a1956e972f388d2045e5b965 WatchSource:0}: Error finding container 324aab5698f88b3132539aa1b04d8abb77dd19d1a1956e972f388d2045e5b965: Status 404 returned error can't find the container with id 324aab5698f88b3132539aa1b04d8abb77dd19d1a1956e972f388d2045e5b965
Nov 03 19:51:13.584504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:13.584466    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:13.584630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:13.584553    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:13.584630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:13.584599    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:13.584963 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:13.584933    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"]
Nov 03 19:51:13.599458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:13.599417    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:14.195606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:14.194331    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" event={"ID":"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce","Type":"ContainerStarted","Data":"cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66"}
Nov 03 19:51:14.195606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:14.194388    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" event={"ID":"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce","Type":"ContainerStarted","Data":"324aab5698f88b3132539aa1b04d8abb77dd19d1a1956e972f388d2045e5b965"}
Nov 03 19:51:14.218706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:14.218651    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" podStartSLOduration=2.218636583 podStartE2EDuration="2.218636583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:14.214059942 +0000 UTC m=+1187.484183239" watchObservedRunningTime="2026-11-03 19:51:14.218636583 +0000 UTC m=+1187.488759879"
Nov 03 19:51:14.529295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:14.529206    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:15.049247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:15.049212    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"]
Nov 03 19:51:15.105328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:15.105251    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9831930f-v2ths"]
Nov 03 19:51:15.224114 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:15.196602    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:15.237652 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:15.237623    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:15.248799 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:15.248772    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"]
Nov 03 19:51:15.252039 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:15.252016    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:15.253086 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:15.253041    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:51:15.257106 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:15.257070    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590f43e5-m2zhf"]
Nov 03 19:51:15.352761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:15.352682    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:15.933629 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:15.933589    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:16.025648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:16.025614    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"]
Nov 03 19:51:16.031338 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:16.031312    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-956950be-zffj7"]
Nov 03 19:51:16.625083 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:16.625047    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:16.805780 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:16.805742    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:16.975096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:16.975058    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432d8dcf-a8c0-4130-8e8e-d86d219b78f7" path="/var/lib/kubelet/pods/432d8dcf-a8c0-4130-8e8e-d86d219b78f7/volumes"
Nov 03 19:51:16.976199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:16.976146    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74aaaa79-143d-4be2-a06d-c304b6c34b5f" path="/var/lib/kubelet/pods/74aaaa79-143d-4be2-a06d-c304b6c34b5f/volumes"
Nov 03 19:51:16.977259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:16.977239    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff" path="/var/lib/kubelet/pods/fd8fa3c8-7ff0-4a06-9ddf-54010de2c1ff/volumes"
Nov 03 19:51:17.459214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:17.455322    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:17.902667 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:17.902627    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:18.064373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:18.064251    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:18.962193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:18.961728    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:19.195347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:19.195310    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"]
Nov 03 19:51:19.201550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:19.201514    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758548f8-trktg"]
Nov 03 19:51:19.745194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:19.732673    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:20.181194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:20.161255    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:20.607714 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:51:20.607608    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:51:20.759197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:20.747516    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:21.054197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:21.046154    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c79918-75c1-40f5-bbfe-5fdcf99f76c5" path="/var/lib/kubelet/pods/24c79918-75c1-40f5-bbfe-5fdcf99f76c5/volumes"
Nov 03 19:51:21.521941 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:21.521896    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:22.242199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:22.228222    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:22.937931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:22.937835    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:23.516013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:23.515973    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"]
Nov 03 19:51:23.527829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:23.527794    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"]
Nov 03 19:51:23.629868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:23.629830    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:23.763046 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:23.763015    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:23.908387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:23.908351    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:24.176531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:24.176437    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"]
Nov 03 19:51:24.260958 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:24.260930    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51db6282-jcslt"]
Nov 03 19:51:24.638246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:24.638204    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:24.864077 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:24.864036    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb955c9-1bfe-49de-99b2-2f3b36f2a066" path="/var/lib/kubelet/pods/8bb955c9-1bfe-49de-99b2-2f3b36f2a066/volumes"
Nov 03 19:51:25.232225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:25.232131    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:25.900194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:25.899095    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:26.066357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:26.066317    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:51:26.066545 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:26.066390    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:26.618678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:26.618635    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:27.370567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:27.369823    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:28.032132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:28.032080    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:28.516499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:28.516457    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m" podUID="961783a9-751a-4630-b785-e43e96ce84b3" containerName="payload-processing" containerID="cri-o://e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0" gracePeriod=25
Nov 03 19:51:28.583410 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:28.528667    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq" podUID="49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f" containerName="payload-pre-processing" containerID="cri-o://ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be" gracePeriod=25
Nov 03 19:51:29.024461 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.022823    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:29.026683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.026657    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:51:29.088456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.088382    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961783a9-751a-4630-b785-e43e96ce84b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961783a9-751a-4630-b785-e43e96ce84b3-plugins-config-volume\") pod \"961783a9-751a-4630-b785-e43e96ce84b3\" (UID: \"961783a9-751a-4630-b785-e43e96ce84b3\") "
Nov 03 19:51:29.088910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.088646    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961783a9-751a-4630-b785-e43e96ce84b3-kube-api-access-x86nl\" (UniqueName: \"kubernetes.io/projected/961783a9-751a-4630-b785-e43e96ce84b3-kube-api-access-x86nl\") pod \"961783a9-751a-4630-b785-e43e96ce84b3\" (UID: \"961783a9-751a-4630-b785-e43e96ce84b3\") "
Nov 03 19:51:29.088910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.088734    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961783a9-751a-4630-b785-e43e96ce84b3-plugins-config-volume" pod "961783a9-751a-4630-b785-e43e96ce84b3" (UID: "961783a9-751a-4630-b785-e43e96ce84b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:29.089644 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.089618    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961783a9-751a-4630-b785-e43e96ce84b3-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:29.090683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.090664    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:51:29.091458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.091435    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961783a9-751a-4630-b785-e43e96ce84b3-kube-api-access-x86nl" pod "961783a9-751a-4630-b785-e43e96ce84b3" (UID: "961783a9-751a-4630-b785-e43e96ce84b3"). InnerVolumeSpecName "kube-api-access-x86nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:29.191298 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.191265    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-plugins-config-volume\") pod \"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f\" (UID: \"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f\") "
Nov 03 19:51:29.191480 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.191309    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-kube-api-access-vlvlv\" (UniqueName: \"kubernetes.io/projected/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-kube-api-access-vlvlv\") pod \"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f\" (UID: \"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f\") "
Nov 03 19:51:29.191679 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.191657    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-plugins-config-volume" pod "49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f" (UID: "49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:29.192153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.192137    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x86nl\" (UniqueName: \"kubernetes.io/projected/961783a9-751a-4630-b785-e43e96ce84b3-kube-api-access-x86nl\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:29.192228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.192156    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:29.193523 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.193503    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-kube-api-access-vlvlv" pod "49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f" (UID: "49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f"). InnerVolumeSpecName "kube-api-access-vlvlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:29.208602 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.208576    4531 generic.go:356] "Generic (PLEG): container finished" podID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerID="246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660" exitCode=0
Nov 03 19:51:29.208748 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.208657    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" event={"ID":"b9597b9a-67e3-492c-afb7-dc2e544b156d","Type":"ContainerDied","Data":"246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660"}
Nov 03 19:51:29.209257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.209232    4531 scope.go:122] "RemoveContainer" containerID="246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660"
Nov 03 19:51:29.210374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.210351    4531 generic.go:356] "Generic (PLEG): container finished" podID="49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f" containerID="ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be" exitCode=0
Nov 03 19:51:29.210473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.210396    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"
Nov 03 19:51:29.210473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.210436    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq" event={"ID":"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f","Type":"ContainerDied","Data":"ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be"}
Nov 03 19:51:29.210569 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.210484    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq" event={"ID":"49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f","Type":"ContainerDied","Data":"ef63c30da5dafbf27c31fad051c7a6be3b13d8bb60edd5a95f127bb1555784d9"}
Nov 03 19:51:29.210569 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.210509    4531 scope.go:122] "RemoveContainer" containerID="ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be"
Nov 03 19:51:29.211385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.211348    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:29.211505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.211437    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:29.211505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.211496    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:29.212348 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.212324    4531 generic.go:356] "Generic (PLEG): container finished" podID="961783a9-751a-4630-b785-e43e96ce84b3" containerID="e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0" exitCode=0
Nov 03 19:51:29.212463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.212352    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m" event={"ID":"961783a9-751a-4630-b785-e43e96ce84b3","Type":"ContainerDied","Data":"e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0"}
Nov 03 19:51:29.212463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.212374    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"
Nov 03 19:51:29.212463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.212403    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m" event={"ID":"961783a9-751a-4630-b785-e43e96ce84b3","Type":"ContainerDied","Data":"f494bc73548142a96ec20782a09dddd2451c3eef57b59b18da8450500a4f2d67"}
Nov 03 19:51:29.219909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.219890    4531 generic.go:356] "Generic (PLEG): container finished" podID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerID="5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d" exitCode=0
Nov 03 19:51:29.220011 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.219946    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" event={"ID":"6ce009d0-6fa8-4449-8517-42a82817a60f","Type":"ContainerDied","Data":"5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d"}
Nov 03 19:51:29.220534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.220516    4531 scope.go:122] "RemoveContainer" containerID="5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d"
Nov 03 19:51:29.225514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.225478    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:29.225594 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.225569    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:29.225637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.225615    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:29.232372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.232350    4531 scope.go:122] "RemoveContainer" containerID="ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be"
Nov 03 19:51:29.232704 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:51:29.232678    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be\": container with ID starting with ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be not found: ID does not exist" containerID="ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be"
Nov 03 19:51:29.232786 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.232715    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be"} err="failed to get container status \"ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be\": rpc error: code = NotFound desc = could not find container \"ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be\": container with ID starting with ee7b0ebe23ea466be9db97146e06d13eec859caa50f90bd4b2d7966230d924be not found: ID does not exist"
Nov 03 19:51:29.232786 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.232737    4531 scope.go:122] "RemoveContainer" containerID="e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0"
Nov 03 19:51:29.279967 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.279632    4531 scope.go:122] "RemoveContainer" containerID="e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0"
Nov 03 19:51:29.280676 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:51:29.280638    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0\": container with ID starting with e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0 not found: ID does not exist" containerID="e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0"
Nov 03 19:51:29.280777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.280684    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0"} err="failed to get container status \"e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0\": rpc error: code = NotFound desc = could not find container \"e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0\": container with ID starting with e36feb8468f462c765feae4cae540a15db52f4220b14484e75ea758cbc4e17d0 not found: ID does not exist"
Nov 03 19:51:29.296016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.295991    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvlv\" (UniqueName: \"kubernetes.io/projected/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f-kube-api-access-vlvlv\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:29.300259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.300237    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"]
Nov 03 19:51:29.307672 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.307652    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c7998-6d4ff44995-p7pfq"]
Nov 03 19:51:29.319699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.318117    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"]
Nov 03 19:51:29.326167 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.323220    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c7998-67bfcdf9b9-dfj8m"]
Nov 03 19:51:29.381490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.381453    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:29.753428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.753391    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"]
Nov 03 19:51:29.754734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.754708    4531 cpu_manager.go:414] "containerMap: removing container" podUID="961783a9-751a-4630-b785-e43e96ce84b3" containerName="payload-processing"
Nov 03 19:51:29.754921 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.754904    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961783a9-751a-4630-b785-e43e96ce84b3" containerName="payload-processing"
Nov 03 19:51:29.755037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.755024    4531 cpu_manager.go:414] "containerMap: removing container" podUID="49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f" containerName="payload-pre-processing"
Nov 03 19:51:29.755700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.755679    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f" containerName="payload-pre-processing"
Nov 03 19:51:29.756215 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.756195    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="961783a9-751a-4630-b785-e43e96ce84b3" containerName="payload-processing"
Nov 03 19:51:29.756338 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.756326    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f" containerName="payload-pre-processing"
Nov 03 19:51:29.769034 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.769011    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"]
Nov 03 19:51:29.769368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.769304    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.792823 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.792793    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-10572d-serving-cert\""
Nov 03 19:51:29.806372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.805636    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.806372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.805705    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b03a364-09e1-4aa1-9644-eaff7261c5dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.806372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.806068    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cdcf\" (UniqueName: \"kubernetes.io/projected/8b03a364-09e1-4aa1-9644-eaff7261c5dd-kube-api-access-2cdcf\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.806372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.806114    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.840478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.840443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:29.955958 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.955752    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cdcf\" (UniqueName: \"kubernetes.io/projected/8b03a364-09e1-4aa1-9644-eaff7261c5dd-kube-api-access-2cdcf\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.956251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.956223    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.956537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.956519    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.957891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.957863    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b03a364-09e1-4aa1-9644-eaff7261c5dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.959990 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.958646    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b03a364-09e1-4aa1-9644-eaff7261c5dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.999234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.982431    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.999234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.990365    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.999234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.991618    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cdcf\" (UniqueName: \"kubernetes.io/projected/8b03a364-09e1-4aa1-9644-eaff7261c5dd-kube-api-access-2cdcf\") pod \"maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:29.999234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:29.992118    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:30.110447 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:30.110352    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:30.129500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:30.129447    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:30.314608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:30.314545    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" event={"ID":"b9597b9a-67e3-492c-afb7-dc2e544b156d","Type":"ContainerStarted","Data":"a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2"}
Nov 03 19:51:30.332586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:30.332381    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" event={"ID":"6ce009d0-6fa8-4449-8517-42a82817a60f","Type":"ContainerStarted","Data":"bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb"}
Nov 03 19:51:30.685647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:30.685613    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:51:30.686071 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:30.686027    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 19:51:30.686319 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:30.686283    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 19:51:31.135097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.133622    4531 patch_prober.go:29] interesting pod/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 19:51:31.135097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.133679    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 19:51:31.226549 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.226510    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:31.244546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.244509    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f" path="/var/lib/kubelet/pods/49cbbfb1-9923-4f2d-96d9-5a5ae9d3231f/volumes"
Nov 03 19:51:31.245754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.245730    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961783a9-751a-4630-b785-e43e96ce84b3" path="/var/lib/kubelet/pods/961783a9-751a-4630-b785-e43e96ce84b3/volumes"
Nov 03 19:51:31.246598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.246580    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:51:31.246806 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.246788    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"]
Nov 03 19:51:31.492277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.484350    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 19:51:31.492277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.484415    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 19:51:31.699197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.692673    4531 scope.go:122] "RemoveContainer" containerID="1dbc92c575684bc26c9d26493ce48c5801264285202f9d38fd50d2f71ef27052"
Nov 03 19:51:31.828367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.828138    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw" event={"ID":"8b03a364-09e1-4aa1-9644-eaff7261c5dd","Type":"ContainerStarted","Data":"c09087a3db901ab6a8d490e3748c7aa11029a5a96b9108e01409b193690babcf"}
Nov 03 19:51:31.835818 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.835794    4531 scope.go:122] "RemoveContainer" containerID="fd64d7710557c4cbbf7bbd647f0e0b5341e056ae5cd376d334112857d74e718a"
Nov 03 19:51:31.908008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:31.907954    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:32.066258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.065795    4531 scope.go:122] "RemoveContainer" containerID="b2d39f64192d294c8e8ff426f2ec83a9609e545648636684278cd16a16d85226"
Nov 03 19:51:32.101382 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:51:32.095998    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbc92c575684bc26c9d26493ce48c5801264285202f9d38fd50d2f71ef27052\": container with ID starting with 1dbc92c575684bc26c9d26493ce48c5801264285202f9d38fd50d2f71ef27052 not found: ID does not exist" containerID="1dbc92c575684bc26c9d26493ce48c5801264285202f9d38fd50d2f71ef27052"
Nov 03 19:51:32.101382 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.096074    4531 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"1dbc92c575684bc26c9d26493ce48c5801264285202f9d38fd50d2f71ef27052\": container with ID starting with 1dbc92c575684bc26c9d26493ce48c5801264285202f9d38fd50d2f71ef27052 not found: ID does not exist"
Nov 03 19:51:32.121689 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:51:32.121652    4531 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"1dbc92c575684bc26c9d26493ce48c5801264285202f9d38fd50d2f71ef27052\": container with ID starting with 1dbc92c575684bc26c9d26493ce48c5801264285202f9d38fd50d2f71ef27052 not found: ID does not exist"
Nov 03 19:51:32.124569 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.124547    4531 scope.go:122] "RemoveContainer" containerID="6d09075dd9c334a28de69858c5b21ddf597cd8bbf3265dd8df8b2a9542513360"
Nov 03 19:51:32.131344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.129796    4531 patch_prober.go:29] interesting pod/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 19:51:32.131344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.129845    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 19:51:32.195910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.195876    4531 scope.go:122] "RemoveContainer" containerID="a6790e969cae82453acbc32397e466a887d714df9d0775458b47192c26971205"
Nov 03 19:51:32.392459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.391908    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 19:51:32.392459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.391975    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 19:51:32.766141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.766104    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:32.962843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.962719    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw" event={"ID":"8b03a364-09e1-4aa1-9644-eaff7261c5dd","Type":"ContainerStarted","Data":"37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901"}
Nov 03 19:51:32.962843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.962796    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:33.000060 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:32.999994    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw" podStartSLOduration=3.128350228 podStartE2EDuration="3.999975196s" totalImagesPullingTime="871.624968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:32.997609545 +0000 UTC m=+1206.267732846" watchObservedRunningTime="2026-11-03 19:51:32.999975196 +0000 UTC m=+1206.270098492"
Nov 03 19:51:33.132275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:33.132239    4531 patch_prober.go:29] interesting pod/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 19:51:33.132561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:33.132528    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 19:51:33.386616 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:33.386513    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 19:51:33.386616 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:33.386582    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 19:51:33.597568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:33.597521    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:33.785261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:33.767074    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:51:34.129332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:34.129298    4531 patch_prober.go:29] interesting pod/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 19:51:34.129499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:34.129357    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 19:51:34.377754 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:34.377710    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:34.382365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:34.382279    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 19:51:34.382632 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:34.382604    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 19:51:35.126262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:35.125692    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:35.146593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:35.145939    4531 patch_prober.go:29] interesting pod/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 19:51:35.146593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:35.146027    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 19:51:35.391199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:35.390642    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 19:51:35.391199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:35.390705    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 19:51:35.391505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:35.391449    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:35.911198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:35.894954    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:36.130395 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:36.130310    4531 patch_prober.go:29] interesting pod/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 19:51:36.131090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:36.131013    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 19:51:36.136211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:36.135526    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"]
Nov 03 19:51:36.212766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:36.212639    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7530c3a7-qdfzz"]
Nov 03 19:51:36.380790 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:36.380754    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 19:51:36.386339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:36.386286    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 19:51:36.615919 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:36.615812    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:36.866671 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:36.866578    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b07052-9a9f-43a5-a99a-47e3a5d13826" path="/var/lib/kubelet/pods/79b07052-9a9f-43a5-a99a-47e3a5d13826/volumes"
Nov 03 19:51:37.030983 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:37.030936    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"]
Nov 03 19:51:37.039784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:37.039728    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d7c820-j72q2"]
Nov 03 19:51:37.134719 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:37.134688    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:51:37.135935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:37.135913    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:51:37.229352 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:37.229315    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:37.395256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:37.394658    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:51:37.410785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:37.396549    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:51:37.413932 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:37.413894    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 19:51:37.901846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:37.901805    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:38.026565 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:38.026533    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"]
Nov 03 19:51:38.033318 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:38.033289    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04da2a04-z7vpq"]
Nov 03 19:51:38.659207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:38.630325    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:38.862555 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:38.862517    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba659e9-0e04-4cd1-87dd-45610cfcafed" path="/var/lib/kubelet/pods/1ba659e9-0e04-4cd1-87dd-45610cfcafed/volumes"
Nov 03 19:51:38.863102 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:38.863087    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f97d12e-2681-4437-aac9-f3f1dd611bf6" path="/var/lib/kubelet/pods/4f97d12e-2681-4437-aac9-f3f1dd611bf6/volumes"
Nov 03 19:51:39.008822 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:39.008738    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 19:51:39.396950 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:39.396891    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:40.137212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:40.137155    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:40.873200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:40.871060    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:41.023295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:41.023264    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"]
Nov 03 19:51:41.028858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:41.028832    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11258e8f-c28tl"]
Nov 03 19:51:41.533201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:41.525891    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:42.229195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:42.222109    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:42.832489 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:42.832451    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:42.930208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:42.928427    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4cacd3-f23c-48bb-b303-1d0773d26915" path="/var/lib/kubelet/pods/aa4cacd3-f23c-48bb-b303-1d0773d26915/volumes"
Nov 03 19:51:43.971236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:43.971137    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:44.716193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:44.715625    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:45.022858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:45.022768    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"]
Nov 03 19:51:45.028536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:45.028507    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b263cacf-zr2x7"]
Nov 03 19:51:45.349021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:45.348802    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:46.163210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:46.152119    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:46.827699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:46.826950    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:46.954201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:46.937773    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9329d4f-1cfe-4868-a5e6-06a841d7dfc2" path="/var/lib/kubelet/pods/d9329d4f-1cfe-4868-a5e6-06a841d7dfc2/volumes"
Nov 03 19:51:46.960249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:46.960209    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:47.024977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:47.024942    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"]
Nov 03 19:51:47.030766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:47.030739    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-871c7fc5-6z8ks"]
Nov 03 19:51:47.618756 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:47.618718    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:47.998290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:47.998256    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:48.022829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:48.022802    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"]
Nov 03 19:51:48.029070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:48.029045    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40185ba6-kwhtp"]
Nov 03 19:51:48.365028 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:48.364912    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:48.511267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:48.511231    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:48.786669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:48.786620    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:48.993842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:48.968868    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a8b817-0777-4330-b177-9c5bc9f3cc64" path="/var/lib/kubelet/pods/33a8b817-0777-4330-b177-9c5bc9f3cc64/volumes"
Nov 03 19:51:48.995109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:48.995070    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964" path="/var/lib/kubelet/pods/7a2a5f8f-d8b8-4c8f-91fb-de16cea6c964/volumes"
Nov 03 19:51:49.290374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:49.290337    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:49.431727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:49.429821    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:50.018855 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:50.018795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:50.022844 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:50.022820    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"]
Nov 03 19:51:50.027685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:50.027662    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09145056-9jsxp"]
Nov 03 19:51:50.574735 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:51:50.574698    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:51:50.689198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:50.686037    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:50.861500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:50.861421    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5861fe0f-fe8a-4704-8a51-689479fac3c9" path="/var/lib/kubelet/pods/5861fe0f-fe8a-4704-8a51-689479fac3c9/volumes"
Nov 03 19:51:51.026246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:51.026168    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"]
Nov 03 19:51:51.031346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:51.031318    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2ea3ef5-cc5cc"]
Nov 03 19:51:51.540731 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:51.540693    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:52.159198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:52.141874    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:52.346243 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:52.346207    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:52.494536 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:52.494448    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:52.796291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:52.796168    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:52.862275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:52.862235    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd166e04-e728-426b-8222-f251ea015d6f" path="/var/lib/kubelet/pods/bd166e04-e728-426b-8222-f251ea015d6f/volumes"
Nov 03 19:51:52.967200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:52.962394    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:53.109275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:53.106752    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:53.361282 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:53.361193    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:53.669872 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:53.669687    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:53.817594 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:53.817561    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:54.751406 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:54.751360    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:56.175397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:56.174127    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:56.341016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:56.340982    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:57.282197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:57.281835    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:58.286252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:58.286214    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:59.061737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.061704    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"]
Nov 03 19:51:59.126437 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.126392    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742b9c55-vzqrl"]
Nov 03 19:51:59.402199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.398891    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:59.402199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.399638    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:51:59.490197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.471706    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:59.615413 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.611284    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Nov 03 19:51:59.615666 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.611327    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:51:59.680255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.663058    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:59.750001 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.722314    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:59.858656 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.847117    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:59.858656 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.847409    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:51:59.896015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.894410    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:51:59.911839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.911796    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:51:59.913367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.913337    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:51:59.915697 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.915664    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:59.963622 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.963520    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:00.020194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:51:59.966995    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:52:00.162426 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.138838    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:00.211634 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.211584    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:00.402060 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.402008    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:00.403767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.403733    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:52:00.405663 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.405639    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:52:00.406921 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.406901    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:52:00.673318 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.666528    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:00.752320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.684185    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.160:8000/ready\": context deadline exceeded"
Nov 03 19:52:00.791920 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.684224    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:52:00.792262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.792239    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:52:00.793748 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.684553    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:00.820188 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.685325    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.160:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:00.821006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.820473    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:52:00.914693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.914639    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:52:00.915496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.915466    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:52:00.917671 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.917525    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:00.918233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.918207    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:52:00.918935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.918889    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:52:00.919104 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.919089    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:52:00.921248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.916856    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:00.921600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.921574    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:52:00.922254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.922231    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:52:00.922342 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.922305    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:52:00.957997 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:00.957893    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:01.283422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:01.283315    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:52:01.405509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:01.405466    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:52:01.500999 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:01.406168    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:52:01.519364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:01.441591    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:01.940198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:01.698616    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:52:02.011206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:01.725949    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:02.011206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:01.753274    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:02.038551 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:01.903130    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:02.062235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:01.981749    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:52:02.062235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:01.981835    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:52:02.062235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:01.981860    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:52:02.087213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.076648    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:02.087213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.076938    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:52:02.088042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.087986    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:52:02.111350 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.111297    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:02.400118 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.377901    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:02.539265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.515508    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:02.539265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.515784    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:02.584251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.584209    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c2ec68-1457-4b91-8efb-14cf29caa92e" path="/var/lib/kubelet/pods/e5c2ec68-1457-4b91-8efb-14cf29caa92e/volumes"
Nov 03 19:52:02.585056 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:52:02.585030    4531 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.652s"
Nov 03 19:52:02.601632 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.601551    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:02.613933 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.613904    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:52:02.616254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.616163    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:52:02.616483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.616468    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:52:02.616618 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.616607    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 19:52:02.616744 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.616735    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:52:02.616854 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.616845    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:52:02.616952 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.616943    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:52:02.617054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.617044    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:52:02.617158 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.617149    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:52:02.617295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.617285    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:52:02.617391 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.617382    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:52:02.617504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.617496    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:52:02.625059 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.625031    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:52:02.703701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:02.703670    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:52:03.122247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:03.122165    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:03.296464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:03.296424    4531 generic.go:356] "Generic (PLEG): container finished" podID="64f7cc83-420d-4739-ace1-db3694d9dca1" containerID="e32c951cc27fb39f598e0a20a20545fcab71436e869dbcc33d28746a18f44944" exitCode=0
Nov 03 19:52:03.296757 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:03.296716    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wfnhm-m7rk8" event={"ID":"64f7cc83-420d-4739-ace1-db3694d9dca1","Type":"ContainerDied","Data":"e32c951cc27fb39f598e0a20a20545fcab71436e869dbcc33d28746a18f44944"}
Nov 03 19:52:03.672080 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:03.672040    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"]
Nov 03 19:52:03.672635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:03.672350    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy" containerID="cri-o://770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b" gracePeriod=30
Nov 03 19:52:03.948074 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:03.947973    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:04.495975 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.495951    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:52:04.604104 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.604011    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f7cc83-420d-4739-ace1-db3694d9dca1-kube-api-access-5kdd6\" (UniqueName: \"kubernetes.io/projected/64f7cc83-420d-4739-ace1-db3694d9dca1-kube-api-access-5kdd6\") pod \"64f7cc83-420d-4739-ace1-db3694d9dca1\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") "
Nov 03 19:52:04.604343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.604317    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64f7cc83-420d-4739-ace1-db3694d9dca1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/64f7cc83-420d-4739-ace1-db3694d9dca1-archives-path\") pod \"64f7cc83-420d-4739-ace1-db3694d9dca1\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") "
Nov 03 19:52:04.604489 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.604408    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-trusted-ca-bundle\") pod \"64f7cc83-420d-4739-ace1-db3694d9dca1\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") "
Nov 03 19:52:04.604593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.604574    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-service-ca-bundle\") pod \"64f7cc83-420d-4739-ace1-db3694d9dca1\" (UID: \"64f7cc83-420d-4739-ace1-db3694d9dca1\") "
Nov 03 19:52:04.604759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.604732    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f7cc83-420d-4739-ace1-db3694d9dca1-archives-path" pod "64f7cc83-420d-4739-ace1-db3694d9dca1" (UID: "64f7cc83-420d-4739-ace1-db3694d9dca1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:04.605149 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.605129    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-trusted-ca-bundle" pod "64f7cc83-420d-4739-ace1-db3694d9dca1" (UID: "64f7cc83-420d-4739-ace1-db3694d9dca1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:04.605288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.605138    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-service-ca-bundle" pod "64f7cc83-420d-4739-ace1-db3694d9dca1" (UID: "64f7cc83-420d-4739-ace1-db3694d9dca1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:04.605422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.605397    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-service-ca-bundle\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:04.605422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.605417    4531 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/64f7cc83-420d-4739-ace1-db3694d9dca1-archives-path\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:04.605544 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.605432    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64f7cc83-420d-4739-ace1-db3694d9dca1-trusted-ca-bundle\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:04.606526 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.606501    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f7cc83-420d-4739-ace1-db3694d9dca1-kube-api-access-5kdd6" pod "64f7cc83-420d-4739-ace1-db3694d9dca1" (UID: "64f7cc83-420d-4739-ace1-db3694d9dca1"). InnerVolumeSpecName "kube-api-access-5kdd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:04.734208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.731532    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kdd6\" (UniqueName: \"kubernetes.io/projected/64f7cc83-420d-4739-ace1-db3694d9dca1-kube-api-access-5kdd6\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:04.743195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:04.736156    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:05.330764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:05.330723    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wfnhm-m7rk8" event={"ID":"64f7cc83-420d-4739-ace1-db3694d9dca1","Type":"ContainerDied","Data":"4840b51a50ba82172249ed358cdfee5dea16e8ba13a2034b4aef0d63d8817ed6"}
Nov 03 19:52:05.330965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:05.330816    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4840b51a50ba82172249ed358cdfee5dea16e8ba13a2034b4aef0d63d8817ed6"
Nov 03 19:52:05.330965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:05.330863    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wfnhm-m7rk8"
Nov 03 19:52:05.428794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:05.428757    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"]
Nov 03 19:52:05.441000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:05.440969    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"]
Nov 03 19:52:05.559817 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:05.559770    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:05.731210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:05.729804    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:06.670663 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:06.670161    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:07.436364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:07.436309    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:08.191661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:08.190688    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:08.871598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:08.871566    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:09.111654 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.111290    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:09.143728 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.143654    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"]
Nov 03 19:52:09.183880 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.183850    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"]
Nov 03 19:52:09.184795 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.184749    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:52:09.189958 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.189935    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f16ed89-89jhh"]
Nov 03 19:52:09.194805 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.194753    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40eec696-fvsvn"]
Nov 03 19:52:09.206504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.206474    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-socket\") pod \"5789cd8c-de84-4996-b0bf-6c72446833db\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") "
Nov 03 19:52:09.206637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.206541    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-kube-api-access-r7wbr\" (UniqueName: \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-kube-api-access-r7wbr\") pod \"5789cd8c-de84-4996-b0bf-6c72446833db\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") "
Nov 03 19:52:09.206772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.206753    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-credential-socket\") pod \"5789cd8c-de84-4996-b0bf-6c72446833db\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") "
Nov 03 19:52:09.206868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.206793    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-envoy\") pod \"5789cd8c-de84-4996-b0bf-6c72446833db\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") "
Nov 03 19:52:09.206868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.206802    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-socket" pod "5789cd8c-de84-4996-b0bf-6c72446833db" (UID: "5789cd8c-de84-4996-b0bf-6c72446833db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:09.206987 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.206879    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-certs\") pod \"5789cd8c-de84-4996-b0bf-6c72446833db\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") "
Nov 03 19:52:09.207039 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.207011    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-istio-token\" (UniqueName: \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-istio-token\") pod \"5789cd8c-de84-4996-b0bf-6c72446833db\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") "
Nov 03 19:52:09.207097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.207038    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-credential-socket" pod "5789cd8c-de84-4996-b0bf-6c72446833db" (UID: "5789cd8c-de84-4996-b0bf-6c72446833db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:09.207167 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.207117    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5789cd8c-de84-4996-b0bf-6c72446833db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5789cd8c-de84-4996-b0bf-6c72446833db-istio-podinfo\") pod \"5789cd8c-de84-4996-b0bf-6c72446833db\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") "
Nov 03 19:52:09.207284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.207242    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-data\") pod \"5789cd8c-de84-4996-b0bf-6c72446833db\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") "
Nov 03 19:52:09.207341 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.207282    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5789cd8c-de84-4996-b0bf-6c72446833db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5789cd8c-de84-4996-b0bf-6c72446833db-istiod-ca-cert\") pod \"5789cd8c-de84-4996-b0bf-6c72446833db\" (UID: \"5789cd8c-de84-4996-b0bf-6c72446833db\") "
Nov 03 19:52:09.207758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.207664    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-data" pod "5789cd8c-de84-4996-b0bf-6c72446833db" (UID: "5789cd8c-de84-4996-b0bf-6c72446833db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:09.207856 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.207832    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5789cd8c-de84-4996-b0bf-6c72446833db-istiod-ca-cert" pod "5789cd8c-de84-4996-b0bf-6c72446833db" (UID: "5789cd8c-de84-4996-b0bf-6c72446833db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:09.208379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.208351    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-certs" pod "5789cd8c-de84-4996-b0bf-6c72446833db" (UID: "5789cd8c-de84-4996-b0bf-6c72446833db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:09.209403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.208920    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:09.209403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.208947    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:09.209403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.208961    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5789cd8c-de84-4996-b0bf-6c72446833db-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:09.209403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.208978    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:09.209403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.208993    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:09.210661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.210623    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-envoy" pod "5789cd8c-de84-4996-b0bf-6c72446833db" (UID: "5789cd8c-de84-4996-b0bf-6c72446833db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:09.210736 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.210667    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-kube-api-access-r7wbr" pod "5789cd8c-de84-4996-b0bf-6c72446833db" (UID: "5789cd8c-de84-4996-b0bf-6c72446833db"). InnerVolumeSpecName "kube-api-access-r7wbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:09.211861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.211837    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5789cd8c-de84-4996-b0bf-6c72446833db-istio-podinfo" pod "5789cd8c-de84-4996-b0bf-6c72446833db" (UID: "5789cd8c-de84-4996-b0bf-6c72446833db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:09.212064 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.212045    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-istio-token" pod "5789cd8c-de84-4996-b0bf-6c72446833db" (UID: "5789cd8c-de84-4996-b0bf-6c72446833db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:09.311212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.311160    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5789cd8c-de84-4996-b0bf-6c72446833db-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:09.311212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.311205    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7wbr\" (UniqueName: \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-kube-api-access-r7wbr\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:09.311212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.311216    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5789cd8c-de84-4996-b0bf-6c72446833db-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:09.311212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.311225    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5789cd8c-de84-4996-b0bf-6c72446833db-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:09.472249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.471564    4531 generic.go:356] "Generic (PLEG): container finished" podID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerID="cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66" exitCode=0
Nov 03 19:52:09.472249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.471819    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" event={"ID":"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce","Type":"ContainerDied","Data":"cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66"}
Nov 03 19:52:09.476193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.472542    4531 scope.go:122] "RemoveContainer" containerID="cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66"
Nov 03 19:52:09.493418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.487706    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:09.493418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.487820    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:09.493418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.487870    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:09.765827 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.765725    4531 generic.go:356] "Generic (PLEG): container finished" podID="5789cd8c-de84-4996-b0bf-6c72446833db" containerID="770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b" exitCode=0
Nov 03 19:52:09.766538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.766145    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"
Nov 03 19:52:09.772636 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.766166    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" event={"ID":"5789cd8c-de84-4996-b0bf-6c72446833db","Type":"ContainerDied","Data":"770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b"}
Nov 03 19:52:09.773482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.773446    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr" event={"ID":"5789cd8c-de84-4996-b0bf-6c72446833db","Type":"ContainerDied","Data":"d4729fc47a7d7681eb6e854511da95d3709c221c63ae3ff5ea97035048335f1a"}
Nov 03 19:52:09.773704 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.773648    4531 scope.go:122] "RemoveContainer" containerID="770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b"
Nov 03 19:52:09.803646 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.803493    4531 scope.go:122] "RemoveContainer" containerID="770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b"
Nov 03 19:52:09.803890 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:52:09.803870    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b\": container with ID starting with 770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b not found: ID does not exist" containerID="770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b"
Nov 03 19:52:09.803967 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.803900    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b"} err="failed to get container status \"770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b\": rpc error: code = NotFound desc = could not find container \"770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b\": container with ID starting with 770bb954d7e104e045759664368c3134e9f63c09eed853a5f0d85db2133c153b not found: ID does not exist"
Nov 03 19:52:09.818672 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.818643    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"]
Nov 03 19:52:09.830206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:09.825832    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-69msr"]
Nov 03 19:52:10.036251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:10.035777    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:10.433203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:10.432245    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8" podUID="df962d0b-648a-4b8c-b894-08ae29ac036e" containerName="payload-processing" containerID="cri-o://84109e8e47f0c9289955ca6fd7e2d233f3d0d7447f85bfd6528bcb58ecf95f96" gracePeriod=25
Nov 03 19:52:10.441904 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:10.441842    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr" podUID="ea449c7e-8da9-4c54-a81a-dfcdff11cef9" containerName="payload-pre-processing" containerID="cri-o://247f5f70a70e5546f6946ca63f64746e68ebe7eaf77046c3652dc0c387b90f38" gracePeriod=25
Nov 03 19:52:10.667053 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:10.666881    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:10.780370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:10.780327    4531 generic.go:356] "Generic (PLEG): container finished" podID="df962d0b-648a-4b8c-b894-08ae29ac036e" containerID="84109e8e47f0c9289955ca6fd7e2d233f3d0d7447f85bfd6528bcb58ecf95f96" exitCode=0
Nov 03 19:52:10.780548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:10.780466    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8" event={"ID":"df962d0b-648a-4b8c-b894-08ae29ac036e","Type":"ContainerDied","Data":"84109e8e47f0c9289955ca6fd7e2d233f3d0d7447f85bfd6528bcb58ecf95f96"}
Nov 03 19:52:10.782680 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:10.782643    4531 generic.go:356] "Generic (PLEG): container finished" podID="ea449c7e-8da9-4c54-a81a-dfcdff11cef9" containerID="247f5f70a70e5546f6946ca63f64746e68ebe7eaf77046c3652dc0c387b90f38" exitCode=0
Nov 03 19:52:10.782815 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:10.782788    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr" event={"ID":"ea449c7e-8da9-4c54-a81a-dfcdff11cef9","Type":"ContainerDied","Data":"247f5f70a70e5546f6946ca63f64746e68ebe7eaf77046c3652dc0c387b90f38"}
Nov 03 19:52:10.785068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:10.785041    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" event={"ID":"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce","Type":"ContainerStarted","Data":"4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651"}
Nov 03 19:52:10.800258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:10.796014    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:11.132210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.121957    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:52:11.132210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.126533    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:52:11.172298 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.171783    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-plugins-config-volume\") pod \"ea449c7e-8da9-4c54-a81a-dfcdff11cef9\" (UID: \"ea449c7e-8da9-4c54-a81a-dfcdff11cef9\") "
Nov 03 19:52:11.172682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.172653    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df962d0b-648a-4b8c-b894-08ae29ac036e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df962d0b-648a-4b8c-b894-08ae29ac036e-plugins-config-volume\") pod \"df962d0b-648a-4b8c-b894-08ae29ac036e\" (UID: \"df962d0b-648a-4b8c-b894-08ae29ac036e\") "
Nov 03 19:52:11.172865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.172850    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-kube-api-access-jndzz\" (UniqueName: \"kubernetes.io/projected/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-kube-api-access-jndzz\") pod \"ea449c7e-8da9-4c54-a81a-dfcdff11cef9\" (UID: \"ea449c7e-8da9-4c54-a81a-dfcdff11cef9\") "
Nov 03 19:52:11.173024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.173011    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df962d0b-648a-4b8c-b894-08ae29ac036e-kube-api-access-m2k8h\" (UniqueName: \"kubernetes.io/projected/df962d0b-648a-4b8c-b894-08ae29ac036e-kube-api-access-m2k8h\") pod \"df962d0b-648a-4b8c-b894-08ae29ac036e\" (UID: \"df962d0b-648a-4b8c-b894-08ae29ac036e\") "
Nov 03 19:52:11.180200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.176643    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df962d0b-648a-4b8c-b894-08ae29ac036e-plugins-config-volume" pod "df962d0b-648a-4b8c-b894-08ae29ac036e" (UID: "df962d0b-648a-4b8c-b894-08ae29ac036e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:11.194207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.193610    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-plugins-config-volume" pod "ea449c7e-8da9-4c54-a81a-dfcdff11cef9" (UID: "ea449c7e-8da9-4c54-a81a-dfcdff11cef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:11.206852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.206788    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df962d0b-648a-4b8c-b894-08ae29ac036e-kube-api-access-m2k8h" pod "df962d0b-648a-4b8c-b894-08ae29ac036e" (UID: "df962d0b-648a-4b8c-b894-08ae29ac036e"). InnerVolumeSpecName "kube-api-access-m2k8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:11.234996 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.234532    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-kube-api-access-jndzz" pod "ea449c7e-8da9-4c54-a81a-dfcdff11cef9" (UID: "ea449c7e-8da9-4c54-a81a-dfcdff11cef9"). InnerVolumeSpecName "kube-api-access-jndzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:11.240088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.239783    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 19:52:11.240088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.239849    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 19:52:11.297852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.294441    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:11.297852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.294481    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df962d0b-648a-4b8c-b894-08ae29ac036e-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:11.297852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.294501    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jndzz\" (UniqueName: \"kubernetes.io/projected/ea449c7e-8da9-4c54-a81a-dfcdff11cef9-kube-api-access-jndzz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:11.297852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.294515    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2k8h\" (UniqueName: \"kubernetes.io/projected/df962d0b-648a-4b8c-b894-08ae29ac036e-kube-api-access-m2k8h\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:11.327349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.327318    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0120cf8e-8eca-48a0-918a-c13fc4baeb05" path="/var/lib/kubelet/pods/0120cf8e-8eca-48a0-918a-c13fc4baeb05/volumes"
Nov 03 19:52:11.328196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.328151    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361de3fa-82d0-408e-b46c-b29a66a68c60" path="/var/lib/kubelet/pods/361de3fa-82d0-408e-b46c-b29a66a68c60/volumes"
Nov 03 19:52:11.328824 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.328803    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" path="/var/lib/kubelet/pods/5789cd8c-de84-4996-b0bf-6c72446833db/volumes"
Nov 03 19:52:11.329606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.329587    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:52:11.486196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.484270    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:11.798813 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.798778    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"
Nov 03 19:52:11.798813 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.798793    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8" event={"ID":"df962d0b-648a-4b8c-b894-08ae29ac036e","Type":"ContainerDied","Data":"c2a018a03164f06bd49b7db264ec8f74656e39117df2722f734aaa277a6b3af0"}
Nov 03 19:52:11.799073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.798879    4531 scope.go:122] "RemoveContainer" containerID="84109e8e47f0c9289955ca6fd7e2d233f3d0d7447f85bfd6528bcb58ecf95f96"
Nov 03 19:52:11.800735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.800706    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr" event={"ID":"ea449c7e-8da9-4c54-a81a-dfcdff11cef9","Type":"ContainerDied","Data":"5e1a0dba56f89088ef6d6ec44ed0ae4a1f63376f07b57e5d5022bbcdf17d217a"}
Nov 03 19:52:11.800884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.800852    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"
Nov 03 19:52:11.831095 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.830892    4531 scope.go:122] "RemoveContainer" containerID="247f5f70a70e5546f6946ca63f64746e68ebe7eaf77046c3652dc0c387b90f38"
Nov 03 19:52:11.868120 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.868054    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"]
Nov 03 19:52:11.875641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.875610    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a536f35b-f874cf565-xkfb8"]
Nov 03 19:52:11.880501 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.880478    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"]
Nov 03 19:52:11.884821 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:11.884795    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a536f35b-6cbd899467-lxcrr"]
Nov 03 19:52:12.182036 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:12.181998    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 19:52:12.182246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:12.182069    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 19:52:12.389199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:12.389080    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:12.873011 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:12.872976    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df962d0b-648a-4b8c-b894-08ae29ac036e" path="/var/lib/kubelet/pods/df962d0b-648a-4b8c-b894-08ae29ac036e/volumes"
Nov 03 19:52:12.873539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:12.873522    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea449c7e-8da9-4c54-a81a-dfcdff11cef9" path="/var/lib/kubelet/pods/ea449c7e-8da9-4c54-a81a-dfcdff11cef9/volumes"
Nov 03 19:52:13.003037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:13.002989    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:13.181648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:13.181602    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 19:52:13.181879 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:13.181682    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 19:52:13.526315 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:13.526218    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:13.748625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:13.748587    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:14.181528 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:14.181492    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 19:52:14.181941 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:14.181551    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 19:52:14.570207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:14.570104    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:15.182149 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:15.182108    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 19:52:15.182627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:15.182218    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 19:52:15.257926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:15.257829    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:15.260483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:15.259411    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:52:15.956021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:15.955979    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:16.107598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:16.107550    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:16.181899 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:16.181866    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 03 19:52:16.182089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:16.181922    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 03 19:52:16.572094 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:16.572054    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:17.270654 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:17.270617    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:52:17.272242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:17.272219    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 19:52:17.339758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:17.339721    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:18.141575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:18.141521    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:18.937422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:18.937384    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:19.787100 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:19.780607    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:20.023270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:20.023231    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"]
Nov 03 19:52:20.026449 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:20.026425    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aec8afb-wnh9d"]
Nov 03 19:52:20.666194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:20.666075    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:20.920047 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:52:20.919953    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:52:20.955767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:20.955723    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a33b82-5427-41cb-9cd9-f2413e4e4dfb" path="/var/lib/kubelet/pods/09a33b82-5427-41cb-9cd9-f2413e4e4dfb/volumes"
Nov 03 19:52:21.085282 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:21.085248    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"]
Nov 03 19:52:21.097548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:21.097414    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53cac197-kjgb6"]
Nov 03 19:52:21.657323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:21.657275    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:22.279198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:22.278805    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.577718 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:22.577601    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:22.604365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:22.578348    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.604883 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:22.586265    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.613984 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:22.613485    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.665877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:22.665288    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.160:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.666467 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:22.666152    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:23.106194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:23.091147    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:23.226194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:23.221787    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:52:23.237332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:23.237288    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:23.318190 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:23.313559    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ca9921-ca34-465e-8f96-cd15ad34e163" path="/var/lib/kubelet/pods/30ca9921-ca34-465e-8f96-cd15ad34e163/volumes"
Nov 03 19:52:24.512024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:24.511967    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:25.452730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:25.452680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:26.225901 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:26.225860    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:26.954214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:26.954160    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:27.596225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:27.596189    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:27.768810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:27.768772    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:28.259534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.259496    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:28.478659 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.478625    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:28.591955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.591875    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"]
Nov 03 19:52:28.593240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593203    4531 cpu_manager.go:414] "containerMap: removing container" podUID="df962d0b-648a-4b8c-b894-08ae29ac036e" containerName="payload-processing"
Nov 03 19:52:28.593240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593227    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df962d0b-648a-4b8c-b894-08ae29ac036e" containerName="payload-processing"
Nov 03 19:52:28.593407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593281    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy"
Nov 03 19:52:28.593407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593290    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy"
Nov 03 19:52:28.593407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593318    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ea449c7e-8da9-4c54-a81a-dfcdff11cef9" containerName="payload-pre-processing"
Nov 03 19:52:28.593407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593326    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea449c7e-8da9-4c54-a81a-dfcdff11cef9" containerName="payload-pre-processing"
Nov 03 19:52:28.593407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593359    4531 cpu_manager.go:414] "containerMap: removing container" podUID="64f7cc83-420d-4739-ace1-db3694d9dca1" containerName="insights-gathering"
Nov 03 19:52:28.593407 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593367    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f7cc83-420d-4739-ace1-db3694d9dca1" containerName="insights-gathering"
Nov 03 19:52:28.593602 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593557    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="df962d0b-648a-4b8c-b894-08ae29ac036e" containerName="payload-processing"
Nov 03 19:52:28.593602 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593575    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5789cd8c-de84-4996-b0bf-6c72446833db" containerName="istio-proxy"
Nov 03 19:52:28.593602 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593585    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea449c7e-8da9-4c54-a81a-dfcdff11cef9" containerName="payload-pre-processing"
Nov 03 19:52:28.593602 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.593600    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f7cc83-420d-4739-ace1-db3694d9dca1" containerName="insights-gathering"
Nov 03 19:52:28.621088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.621049    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.622338 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.621838    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"]
Nov 03 19:52:28.627127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.627082    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a9cd5e-openshift-default-dockercfg-p74d6\""
Nov 03 19:52:28.678129 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.678095    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:28.767202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.766309    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-envoy\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.767202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.766397    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istiod-ca-cert\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.767202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.766520    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-certs\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.767202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.766563    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-podinfo\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.767202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.766626    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4q8v\" (UniqueName: \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-kube-api-access-f4q8v\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.767202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.766743    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-credential-socket\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.767202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.766813    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-socket\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.767202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.767111    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-data\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.767202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.767161    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-token\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.828209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.827070    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.890488    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-credential-socket\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.890579    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-socket\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.890858    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-data\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.890903    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-token\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.891001    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-envoy\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.891059    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istiod-ca-cert\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.891167    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-certs\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.891224    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-podinfo\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.891290    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4q8v\" (UniqueName: \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-kube-api-access-f4q8v\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.892153    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-credential-socket\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.892439    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-socket\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.905351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.892671    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-data\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.910191 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.908336    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-certs\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.910191 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.908972    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istiod-ca-cert\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.940199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.939062    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-token\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.951200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.942649    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-envoy\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.974687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.952458    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4q8v\" (UniqueName: \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-kube-api-access-f4q8v\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:28.974687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:28.974152    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-podinfo\") pod \"e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") " pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:29.239232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:29.239074    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:29.250814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:29.247569    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:29.658014 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:29.657990    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"]
Nov 03 19:52:29.658385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:29.658300    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:29.658385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:29.658362    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:29.658464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:29.658393    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:30.020255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:30.017722    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:30.202963 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:30.202923    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" event={"ID":"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d","Type":"ContainerStarted","Data":"d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168"}
Nov 03 19:52:30.203136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:30.202984    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" event={"ID":"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d","Type":"ContainerStarted","Data":"bd4ee1ff50953afebbfa450fdb7df6311ee5a10647b81d83d54752d5b51fde92"}
Nov 03 19:52:30.223694 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:30.223636    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" podStartSLOduration=2.223616854 podStartE2EDuration="2.223616854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:30.221259192 +0000 UTC m=+1263.491382543" watchObservedRunningTime="2026-11-03 19:52:30.223616854 +0000 UTC m=+1263.493740149"
Nov 03 19:52:30.248933 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:30.248898    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:30.255857 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:30.255829    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:30.788869 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:30.788826    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:31.214543 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:31.214510    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:31.215619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:31.215599    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:52:31.434582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:31.434539    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:32.303077 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:32.300239    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:32.406877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:32.406840    4531 scope.go:122] "RemoveContainer" containerID="ee051535539bf1382884b1a5d23e4a576d5d47f08721cb9720874001c0c81287"
Nov 03 19:52:32.442611 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:32.442583    4531 scope.go:122] "RemoveContainer" containerID="0df56b2669aa99c3580402e4d9e07ba4c81c8448e39ac77a8f2b7515f2b66b79"
Nov 03 19:52:32.454128 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:32.454102    4531 scope.go:122] "RemoveContainer" containerID="98d7af69bc6817a24d3164ed4a543def2eb5ab9ec269c453552be626ab7f5d44"
Nov 03 19:52:32.465163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:32.465141    4531 scope.go:122] "RemoveContainer" containerID="3883c45641929d03ef302a74242192571cdf683fc2654cf78db7a29f3d378251"
Nov 03 19:52:32.476063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:32.476035    4531 scope.go:122] "RemoveContainer" containerID="02de99a85d61c171cd9ce4c9b1af8c0b7d870b0b20523fd2033ceca8d26a2a2f"
Nov 03 19:52:32.486132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:32.486111    4531 scope.go:122] "RemoveContainer" containerID="4030faad0f0d8af23a6de3daf65cbe39942cf9e0784116843b91e2d2fe634f6e"
Nov 03 19:52:32.496464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:32.496443    4531 scope.go:122] "RemoveContainer" containerID="c22985759731f58419dee6dc1ba1293eeb027fa8506c6c35310114d43eb2d1b2"
Nov 03 19:52:32.500830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:32.500802    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:32.508417 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:32.508394    4531 scope.go:122] "RemoveContainer" containerID="b1af3cfa42e3ae01a30b206e671cb4201f4fc9041c50d83661350b5a149ce0e0"
Nov 03 19:52:32.700121 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:32.700053    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:33.359198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:33.356691    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:33.470137 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:33.469698    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"]
Nov 03 19:52:33.470137 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:33.469968    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy" containerID="cri-o://daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464" gracePeriod=30
Nov 03 19:52:33.792601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:33.792566    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:33.929248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:33.929215    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:34.714210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:34.714154    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:35.424638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:35.424597    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:36.043719 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:36.043683    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:36.561531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:36.561499    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:36.742590 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:36.740456    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:37.135339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:37.135262    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:37.292834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:37.292746    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:37.590658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:37.590552    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:38.211384 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:38.211335    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:38.481852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:38.481753    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:38.665160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:38.665022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:38.821237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:38.815133    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:38.963276 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:38.956891    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:39.114026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.113932    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:39.315314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.315286    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:52:39.390914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.390883    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-kube-api-access-565dx\" (UniqueName: \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-kube-api-access-565dx\") pod \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") "
Nov 03 19:52:39.391095 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.390924    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-certs\") pod \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") "
Nov 03 19:52:39.391095 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391026    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-podinfo\") pod \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") "
Nov 03 19:52:39.391095 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391083    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-data\") pod \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") "
Nov 03 19:52:39.391292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391255    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istiod-ca-cert\") pod \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") "
Nov 03 19:52:39.391380 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391359    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-certs" pod "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" (UID: "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:39.391476 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391452    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-data" pod "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" (UID: "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:39.391579 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391466    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-socket\") pod \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") "
Nov 03 19:52:39.391675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391652    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-credential-socket\") pod \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") "
Nov 03 19:52:39.391675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391651    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istiod-ca-cert" pod "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" (UID: "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:39.391843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391729    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-socket" pod "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" (UID: "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:39.391843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391748    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-envoy\") pod \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") "
Nov 03 19:52:39.391993 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391922    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-credential-socket" pod "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" (UID: "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:39.391993 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.391941    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-token\" (UniqueName: \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-token\") pod \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\" (UID: \"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7\") "
Nov 03 19:52:39.392835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.392812    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:39.392983 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.392837    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:39.392983 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.392853    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:39.392983 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.392867    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:39.392983 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.392879    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:39.394097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.394065    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-kube-api-access-565dx" pod "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" (UID: "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7"). InnerVolumeSpecName "kube-api-access-565dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:39.394270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.394189    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-envoy" pod "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" (UID: "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:39.394472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.394446    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-podinfo" pod "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" (UID: "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:39.394668 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.394645    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-token" pod "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" (UID: "3fb82e5d-855d-4e5b-a3aa-b979988aa2d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:39.405872 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.405841    4531 generic.go:356] "Generic (PLEG): container finished" podID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerID="daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464" exitCode=0
Nov 03 19:52:39.405970 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.405887    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"
Nov 03 19:52:39.405970 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.405923    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" event={"ID":"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7","Type":"ContainerDied","Data":"daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464"}
Nov 03 19:52:39.406073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.405985    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf" event={"ID":"3fb82e5d-855d-4e5b-a3aa-b979988aa2d7","Type":"ContainerDied","Data":"e2bdaed4451b45de2369476555dfc08a7ca50ec1372203671d40cb863e3d151a"}
Nov 03 19:52:39.406073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.406013    4531 scope.go:122] "RemoveContainer" containerID="daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464"
Nov 03 19:52:39.438663 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.438560    4531 scope.go:122] "RemoveContainer" containerID="daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464"
Nov 03 19:52:39.438882 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:52:39.438861    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464\": container with ID starting with daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464 not found: ID does not exist" containerID="daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464"
Nov 03 19:52:39.438958 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.438895    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464"} err="failed to get container status \"daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464\": rpc error: code = NotFound desc = could not find container \"daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464\": container with ID starting with daf23dcc9de7bb9a51327cb52314f2c1c864f620ee650898971b426be2e26464 not found: ID does not exist"
Nov 03 19:52:39.473065 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.473020    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"]
Nov 03 19:52:39.478806 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.478775    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-7ttwf"]
Nov 03 19:52:39.494541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.494513    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:39.494541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.494539    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:39.494720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.494549    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-565dx\" (UniqueName: \"kubernetes.io/projected/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-kube-api-access-565dx\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:39.494720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.494559    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:39.760553 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:39.746659    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:40.437209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:40.436110    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:40.862073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:40.861976    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" path="/var/lib/kubelet/pods/3fb82e5d-855d-4e5b-a3aa-b979988aa2d7/volumes"
Nov 03 19:52:41.040076 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:41.040040    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:41.656189 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:41.656135    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:42.322430 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:42.322390    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:42.507161 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:42.507113    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"]
Nov 03 19:52:42.634270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:42.634234    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:43.044045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:43.043952    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:43.644597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:43.644554    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:44.658203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:44.646828    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:45.265105 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:45.265062    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:45.455496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:45.455460    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:45.592574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:45.592455    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"]
Nov 03 19:52:45.593019 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:45.592974    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy" containerID="cri-o://bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb" gracePeriod=30
Nov 03 19:52:45.664339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:45.664303    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:46.383202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:46.378848    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:46.522530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:46.522476    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:46.824074 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:46.823971    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:46.960302 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:46.960253    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:47.657047 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:47.646972    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:47.943709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:47.943616    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:48.355979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:48.355892    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:48.558204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:48.552800    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:48.987151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:48.986862    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"]
Nov 03 19:52:48.987151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:48.986923    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"]
Nov 03 19:52:48.987151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:48.987114    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerName="istio-proxy" containerID="cri-o://bb710b5d10d4db0b0711e14186983e0cb5fa2d6a7b089a2bc777ec9487f8f643" gracePeriod=30
Nov 03 19:52:48.988762 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:48.988507    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" containerID="cri-o://492e546580bad772080d5d7e0feb0225f8bdd020d4d014d7fe163122f129dcb1" gracePeriod=30
Nov 03 19:52:49.015279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:49.015246    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:52:49.015418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:49.015305    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:52:49.494885 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:49.494601    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:50.453198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.431602    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:50.584610 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:52:50.584567    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:52:50.632110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.632079    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:50.764620 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.764584    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:50.903634 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.903605    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:52:50.960487 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.960451    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-socket\") pod \"6ce009d0-6fa8-4449-8517-42a82817a60f\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") "
Nov 03 19:52:50.960699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.960588    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-data\") pod \"6ce009d0-6fa8-4449-8517-42a82817a60f\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") "
Nov 03 19:52:50.960699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.960647    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-podinfo\") pod \"6ce009d0-6fa8-4449-8517-42a82817a60f\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") "
Nov 03 19:52:50.960828 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.960746    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-kube-api-access-r62qc\" (UniqueName: \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-kube-api-access-r62qc\") pod \"6ce009d0-6fa8-4449-8517-42a82817a60f\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") "
Nov 03 19:52:50.960890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.960829    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-token\" (UniqueName: \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-token\") pod \"6ce009d0-6fa8-4449-8517-42a82817a60f\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") "
Nov 03 19:52:50.960890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.960835    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-socket" pod "6ce009d0-6fa8-4449-8517-42a82817a60f" (UID: "6ce009d0-6fa8-4449-8517-42a82817a60f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:50.960890 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.960870    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ce009d0-6fa8-4449-8517-42a82817a60f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ce009d0-6fa8-4449-8517-42a82817a60f-istiod-ca-cert\") pod \"6ce009d0-6fa8-4449-8517-42a82817a60f\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") "
Nov 03 19:52:50.961045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.960999    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-credential-socket\") pod \"6ce009d0-6fa8-4449-8517-42a82817a60f\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") "
Nov 03 19:52:50.961134 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.961116    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-certs\") pod \"6ce009d0-6fa8-4449-8517-42a82817a60f\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") "
Nov 03 19:52:50.961261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.961244    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-envoy\") pod \"6ce009d0-6fa8-4449-8517-42a82817a60f\" (UID: \"6ce009d0-6fa8-4449-8517-42a82817a60f\") "
Nov 03 19:52:50.961764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.961740    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-credential-socket" pod "6ce009d0-6fa8-4449-8517-42a82817a60f" (UID: "6ce009d0-6fa8-4449-8517-42a82817a60f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:50.962371 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.962347    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-data" pod "6ce009d0-6fa8-4449-8517-42a82817a60f" (UID: "6ce009d0-6fa8-4449-8517-42a82817a60f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:50.962482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.961756    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-certs" pod "6ce009d0-6fa8-4449-8517-42a82817a60f" (UID: "6ce009d0-6fa8-4449-8517-42a82817a60f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:50.962674 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.962650    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce009d0-6fa8-4449-8517-42a82817a60f-istiod-ca-cert" pod "6ce009d0-6fa8-4449-8517-42a82817a60f" (UID: "6ce009d0-6fa8-4449-8517-42a82817a60f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:50.963098 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.963075    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:50.963254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.963237    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:50.963378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.963355    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:50.963476 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.963381    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:50.963476 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.963396    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ce009d0-6fa8-4449-8517-42a82817a60f-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:50.964882 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.964849    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-token" pod "6ce009d0-6fa8-4449-8517-42a82817a60f" (UID: "6ce009d0-6fa8-4449-8517-42a82817a60f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:50.964988 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.964928    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-envoy" pod "6ce009d0-6fa8-4449-8517-42a82817a60f" (UID: "6ce009d0-6fa8-4449-8517-42a82817a60f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:50.966031 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.966000    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-kube-api-access-r62qc" pod "6ce009d0-6fa8-4449-8517-42a82817a60f" (UID: "6ce009d0-6fa8-4449-8517-42a82817a60f"). InnerVolumeSpecName "kube-api-access-r62qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:50.966234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:50.966208    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-podinfo" pod "6ce009d0-6fa8-4449-8517-42a82817a60f" (UID: "6ce009d0-6fa8-4449-8517-42a82817a60f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:51.067039 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.067002    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:51.067039 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.067041    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r62qc\" (UniqueName: \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-kube-api-access-r62qc\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:51.067528 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.067057    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:51.067528 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.067070    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ce009d0-6fa8-4449-8517-42a82817a60f-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:51.270917 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.270839    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:51.526764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.526671    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:51.683482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.681334    4531 generic.go:356] "Generic (PLEG): container finished" podID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerID="bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb" exitCode=0
Nov 03 19:52:51.683482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.681468    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" event={"ID":"6ce009d0-6fa8-4449-8517-42a82817a60f","Type":"ContainerDied","Data":"bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb"}
Nov 03 19:52:51.683482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.681532    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw" event={"ID":"6ce009d0-6fa8-4449-8517-42a82817a60f","Type":"ContainerDied","Data":"6f26b7d40a635fd0a051bd7dd9e5eedba27176135e4b5519ece2d8b59b02d781"}
Nov 03 19:52:51.683482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.681603    4531 scope.go:122] "RemoveContainer" containerID="bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb"
Nov 03 19:52:51.683482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.681795    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"
Nov 03 19:52:51.818891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.818860    4531 scope.go:122] "RemoveContainer" containerID="5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d"
Nov 03 19:52:51.831254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.831195    4531 scope.go:122] "RemoveContainer" containerID="bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb"
Nov 03 19:52:51.831554 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:52:51.831532    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb\": container with ID starting with bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb not found: ID does not exist" containerID="bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb"
Nov 03 19:52:51.831627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.831563    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb"} err="failed to get container status \"bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb\": rpc error: code = NotFound desc = could not find container \"bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb\": container with ID starting with bc2fe9f786790eda7792310cbbff8b852db23c9aafd34e90864a87ffeb88d1eb not found: ID does not exist"
Nov 03 19:52:51.831627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.831583    4531 scope.go:122] "RemoveContainer" containerID="5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d"
Nov 03 19:52:51.831877 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:52:51.831856    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d\": container with ID starting with 5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d not found: ID does not exist" containerID="5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d"
Nov 03 19:52:51.831926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.831886    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d"} err="failed to get container status \"5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d\": rpc error: code = NotFound desc = could not find container \"5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d\": container with ID starting with 5235938b28f9e3860523c32667ba9adc75cd902dc14fa5f5d167535825a94a5d not found: ID does not exist"
Nov 03 19:52:51.838383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.838356    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"]
Nov 03 19:52:51.843002 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:51.842979    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-82b108-openshift-default-d596c9f89-9p7pw"]
Nov 03 19:52:52.489461 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.489430    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:52.558905 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.558873    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"]
Nov 03 19:52:52.560931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.560901    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy"
Nov 03 19:52:52.560931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.560929    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy"
Nov 03 19:52:52.561131 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.560950    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy"
Nov 03 19:52:52.561131 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.560959    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy"
Nov 03 19:52:52.561131 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.560988    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy"
Nov 03 19:52:52.561131 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.560997    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy"
Nov 03 19:52:52.561368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.561232    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy"
Nov 03 19:52:52.561368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.561246    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb82e5d-855d-4e5b-a3aa-b979988aa2d7" containerName="istio-proxy"
Nov 03 19:52:52.561368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.561257    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" containerName="istio-proxy"
Nov 03 19:52:52.568235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.568209    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"]
Nov 03 19:52:52.568427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.568273    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 19:52:52.576234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.574927    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"]
Nov 03 19:52:52.613200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.604465    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"]
Nov 03 19:52:52.613200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.604666    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 19:52:52.624826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.624224    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bdb694\""
Nov 03 19:52:52.624826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.624706    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bdb694-dockercfg-54rpv\""
Nov 03 19:52:52.654905 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.646850    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:52.778542 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.765260    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72xp6\" (UniqueName: \"kubernetes.io/projected/496775e8-474f-4196-a144-4a8526f49bbe-kube-api-access-72xp6\") pod \"payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk\" (UID: \"496775e8-474f-4196-a144-4a8526f49bbe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 19:52:52.809672 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.809577    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg7hn\" (UniqueName: \"kubernetes.io/projected/0951ddac-5219-4bc1-951b-21aece6dc501-kube-api-access-hg7hn\") pod \"payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds\" (UID: \"0951ddac-5219-4bc1-951b-21aece6dc501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 19:52:52.810826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.810787    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496775e8-474f-4196-a144-4a8526f49bbe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk\" (UID: \"496775e8-474f-4196-a144-4a8526f49bbe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 19:52:52.811328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.811307    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0951ddac-5219-4bc1-951b-21aece6dc501-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds\" (UID: \"0951ddac-5219-4bc1-951b-21aece6dc501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 19:52:52.837720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.837681    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:52.925374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.925339    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496775e8-474f-4196-a144-4a8526f49bbe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk\" (UID: \"496775e8-474f-4196-a144-4a8526f49bbe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 19:52:52.925726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.925698    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0951ddac-5219-4bc1-951b-21aece6dc501-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds\" (UID: \"0951ddac-5219-4bc1-951b-21aece6dc501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 19:52:52.926037 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.926017    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72xp6\" (UniqueName: \"kubernetes.io/projected/496775e8-474f-4196-a144-4a8526f49bbe-kube-api-access-72xp6\") pod \"payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk\" (UID: \"496775e8-474f-4196-a144-4a8526f49bbe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 19:52:52.926267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.926248    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg7hn\" (UniqueName: \"kubernetes.io/projected/0951ddac-5219-4bc1-951b-21aece6dc501-kube-api-access-hg7hn\") pod \"payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds\" (UID: \"0951ddac-5219-4bc1-951b-21aece6dc501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 19:52:52.927696 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.927671    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496775e8-474f-4196-a144-4a8526f49bbe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk\" (UID: \"496775e8-474f-4196-a144-4a8526f49bbe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 19:52:52.928484 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.928465    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0951ddac-5219-4bc1-951b-21aece6dc501-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds\" (UID: \"0951ddac-5219-4bc1-951b-21aece6dc501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 19:52:52.959195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.951934    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg7hn\" (UniqueName: \"kubernetes.io/projected/0951ddac-5219-4bc1-951b-21aece6dc501-kube-api-access-hg7hn\") pod \"payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds\" (UID: \"0951ddac-5219-4bc1-951b-21aece6dc501\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 19:52:52.960930 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.960894    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ce009d0-6fa8-4449-8517-42a82817a60f" path="/var/lib/kubelet/pods/6ce009d0-6fa8-4449-8517-42a82817a60f/volumes"
Nov 03 19:52:52.963400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.961192    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 19:52:52.968192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:52.965845    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72xp6\" (UniqueName: \"kubernetes.io/projected/496775e8-474f-4196-a144-4a8526f49bbe-kube-api-access-72xp6\") pod \"payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk\" (UID: \"496775e8-474f-4196-a144-4a8526f49bbe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 19:52:53.262822 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.261805    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 19:52:53.452605 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:52:53.452573    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0951ddac_5219_4bc1_951b_21aece6dc501.slice/crio-246b7d9da416abc6be6d9ada3ed69561aad96311230e705d8ff99e4536ce3b46 WatchSource:0}: Error finding container 246b7d9da416abc6be6d9ada3ed69561aad96311230e705d8ff99e4536ce3b46: Status 404 returned error can't find the container with id 246b7d9da416abc6be6d9ada3ed69561aad96311230e705d8ff99e4536ce3b46
Nov 03 19:52:53.452736 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.452659    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"]
Nov 03 19:52:53.566975 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.566944    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"]
Nov 03 19:52:53.567327 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:52:53.567113    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496775e8_474f_4196_a144_4a8526f49bbe.slice/crio-106395ad7c7df3e35e90c934fc43506710ef169e271fa69204ec4e3b4863d4e9 WatchSource:0}: Error finding container 106395ad7c7df3e35e90c934fc43506710ef169e271fa69204ec4e3b4863d4e9: Status 404 returned error can't find the container with id 106395ad7c7df3e35e90c934fc43506710ef169e271fa69204ec4e3b4863d4e9
Nov 03 19:52:53.703974 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.703937    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk" event={"ID":"496775e8-474f-4196-a144-4a8526f49bbe","Type":"ContainerStarted","Data":"7c12a04aa5a395aa23b299e1a116a064c48390e482442e6e2bf2923f8f675dae"}
Nov 03 19:52:53.704156 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.704010    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk" event={"ID":"496775e8-474f-4196-a144-4a8526f49bbe","Type":"ContainerStarted","Data":"106395ad7c7df3e35e90c934fc43506710ef169e271fa69204ec4e3b4863d4e9"}
Nov 03 19:52:53.704156 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.704035    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 19:52:53.706558 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.706231    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds" event={"ID":"0951ddac-5219-4bc1-951b-21aece6dc501","Type":"ContainerStarted","Data":"719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1"}
Nov 03 19:52:53.706558 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.706286    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds" event={"ID":"0951ddac-5219-4bc1-951b-21aece6dc501","Type":"ContainerStarted","Data":"246b7d9da416abc6be6d9ada3ed69561aad96311230e705d8ff99e4536ce3b46"}
Nov 03 19:52:53.706558 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.706400    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 19:52:53.746212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.741350    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:53.795464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.795391    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk" podStartSLOduration=1.795371422 podStartE2EDuration="1.795371422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:53.761640718 +0000 UTC m=+1287.031764013" watchObservedRunningTime="2026-11-03 19:52:53.795371422 +0000 UTC m=+1287.065494718"
Nov 03 19:52:53.823565 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.823478    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds" podStartSLOduration=1.823454242 podStartE2EDuration="1.823454242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:53.784590279 +0000 UTC m=+1287.054713575" watchObservedRunningTime="2026-11-03 19:52:53.823454242 +0000 UTC m=+1287.093577538"
Nov 03 19:52:53.973658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.973624    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:53.984357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.984315    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"]
Nov 03 19:52:53.989525 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.989495    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 19:52:53.992780 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.992694    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-9f3188-dockercfg-npkrt\""
Nov 03 19:52:53.992780 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.992760    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-9f3188\""
Nov 03 19:52:53.997275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:53.997252    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"]
Nov 03 19:52:54.048720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.048687    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"]
Nov 03 19:52:54.068240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.061975    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"]
Nov 03 19:52:54.068240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.062101    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 19:52:54.139214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.103243    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73377b59-5b9c-4cb9-8c69-413b5c215d30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r\" (UID: \"73377b59-5b9c-4cb9-8c69-413b5c215d30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 19:52:54.139214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.103474    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vgn\" (UniqueName: \"kubernetes.io/projected/73377b59-5b9c-4cb9-8c69-413b5c215d30-kube-api-access-q5vgn\") pod \"payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r\" (UID: \"73377b59-5b9c-4cb9-8c69-413b5c215d30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 19:52:54.176972 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.176933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:54.444213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.422448    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73377b59-5b9c-4cb9-8c69-413b5c215d30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r\" (UID: \"73377b59-5b9c-4cb9-8c69-413b5c215d30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 19:52:54.444213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.422672    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vgn\" (UniqueName: \"kubernetes.io/projected/73377b59-5b9c-4cb9-8c69-413b5c215d30-kube-api-access-q5vgn\") pod \"payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r\" (UID: \"73377b59-5b9c-4cb9-8c69-413b5c215d30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 19:52:54.444213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.424159    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73377b59-5b9c-4cb9-8c69-413b5c215d30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r\" (UID: \"73377b59-5b9c-4cb9-8c69-413b5c215d30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 19:52:54.462317 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.451041    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"]
Nov 03 19:52:54.580195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.570749    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vgn\" (UniqueName: \"kubernetes.io/projected/73377b59-5b9c-4cb9-8c69-413b5c215d30-kube-api-access-q5vgn\") pod \"payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r\" (UID: \"73377b59-5b9c-4cb9-8c69-413b5c215d30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 19:52:54.580195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.540337    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksq5m\" (UniqueName: \"kubernetes.io/projected/01880166-1848-4cae-a23b-8ffc842ed06c-kube-api-access-ksq5m\") pod \"payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq\" (UID: \"01880166-1848-4cae-a23b-8ffc842ed06c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 19:52:54.580195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.576760    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01880166-1848-4cae-a23b-8ffc842ed06c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq\" (UID: \"01880166-1848-4cae-a23b-8ffc842ed06c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 19:52:54.589839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.589801    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:54.683194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.681688    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksq5m\" (UniqueName: \"kubernetes.io/projected/01880166-1848-4cae-a23b-8ffc842ed06c-kube-api-access-ksq5m\") pod \"payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq\" (UID: \"01880166-1848-4cae-a23b-8ffc842ed06c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 19:52:54.683194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.682056    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01880166-1848-4cae-a23b-8ffc842ed06c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq\" (UID: \"01880166-1848-4cae-a23b-8ffc842ed06c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 19:52:54.687275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.683726    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01880166-1848-4cae-a23b-8ffc842ed06c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq\" (UID: \"01880166-1848-4cae-a23b-8ffc842ed06c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 19:52:54.693961 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.693932    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksq5m\" (UniqueName: \"kubernetes.io/projected/01880166-1848-4cae-a23b-8ffc842ed06c-kube-api-access-ksq5m\") pod \"payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq\" (UID: \"01880166-1848-4cae-a23b-8ffc842ed06c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 19:52:54.726714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.726564    4531 generic.go:356] "Generic (PLEG): container finished" podID="645da40a-5b03-4200-adba-b2da577b27f3" containerID="492e546580bad772080d5d7e0feb0225f8bdd020d4d014d7fe163122f129dcb1" exitCode=0
Nov 03 19:52:54.727111 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.727074    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" event={"ID":"645da40a-5b03-4200-adba-b2da577b27f3","Type":"ContainerDied","Data":"492e546580bad772080d5d7e0feb0225f8bdd020d4d014d7fe163122f129dcb1"}
Nov 03 19:52:54.730194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.729686    4531 generic.go:356] "Generic (PLEG): container finished" podID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerID="bb710b5d10d4db0b0711e14186983e0cb5fa2d6a7b089a2bc777ec9487f8f643" exitCode=0
Nov 03 19:52:54.731215 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.731151    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" event={"ID":"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04","Type":"ContainerDied","Data":"bb710b5d10d4db0b0711e14186983e0cb5fa2d6a7b089a2bc777ec9487f8f643"}
Nov 03 19:52:54.832498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.832465    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:52:54.898050 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.898013    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 19:52:54.933205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:54.903730    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.003011    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-socket\") pod \"645da40a-5b03-4200-adba-b2da577b27f3\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") "
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.003121    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-istio-token\" (UniqueName: \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-istio-token\") pod \"645da40a-5b03-4200-adba-b2da577b27f3\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") "
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.003196    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-envoy\") pod \"645da40a-5b03-4200-adba-b2da577b27f3\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") "
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.003427    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645da40a-5b03-4200-adba-b2da577b27f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645da40a-5b03-4200-adba-b2da577b27f3-istiod-ca-cert\") pod \"645da40a-5b03-4200-adba-b2da577b27f3\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") "
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.003525    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-kube-api-access-z8j75\" (UniqueName: \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-kube-api-access-z8j75\") pod \"645da40a-5b03-4200-adba-b2da577b27f3\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") "
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.003578    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-certs\") pod \"645da40a-5b03-4200-adba-b2da577b27f3\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") "
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.003612    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-data\") pod \"645da40a-5b03-4200-adba-b2da577b27f3\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") "
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.003641    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/645da40a-5b03-4200-adba-b2da577b27f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645da40a-5b03-4200-adba-b2da577b27f3-istio-podinfo\") pod \"645da40a-5b03-4200-adba-b2da577b27f3\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") "
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.003805    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-credential-socket\") pod \"645da40a-5b03-4200-adba-b2da577b27f3\" (UID: \"645da40a-5b03-4200-adba-b2da577b27f3\") "
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.005478    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-data" pod "645da40a-5b03-4200-adba-b2da577b27f3" (UID: "645da40a-5b03-4200-adba-b2da577b27f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.029412    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645da40a-5b03-4200-adba-b2da577b27f3-istiod-ca-cert" pod "645da40a-5b03-4200-adba-b2da577b27f3" (UID: "645da40a-5b03-4200-adba-b2da577b27f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.030349    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-istio-token" pod "645da40a-5b03-4200-adba-b2da577b27f3" (UID: "645da40a-5b03-4200-adba-b2da577b27f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.042365    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-socket" pod "645da40a-5b03-4200-adba-b2da577b27f3" (UID: "645da40a-5b03-4200-adba-b2da577b27f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.062508    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-certs" pod "645da40a-5b03-4200-adba-b2da577b27f3" (UID: "645da40a-5b03-4200-adba-b2da577b27f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.081612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.064492    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-credential-socket" pod "645da40a-5b03-4200-adba-b2da577b27f3" (UID: "645da40a-5b03-4200-adba-b2da577b27f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.142024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.086044    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-kube-api-access-z8j75" pod "645da40a-5b03-4200-adba-b2da577b27f3" (UID: "645da40a-5b03-4200-adba-b2da577b27f3"). InnerVolumeSpecName "kube-api-access-z8j75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:55.142024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.086571    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-envoy" pod "645da40a-5b03-4200-adba-b2da577b27f3" (UID: "645da40a-5b03-4200-adba-b2da577b27f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.177504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.170779    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/645da40a-5b03-4200-adba-b2da577b27f3-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.177504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.170820    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8j75\" (UniqueName: \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-kube-api-access-z8j75\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.177504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.170839    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.177504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.170851    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.177504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.170865    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.177504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.170878    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.177504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.170891    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/645da40a-5b03-4200-adba-b2da577b27f3-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.177504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.170904    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/645da40a-5b03-4200-adba-b2da577b27f3-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.301638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.198464    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/645da40a-5b03-4200-adba-b2da577b27f3-istio-podinfo" pod "645da40a-5b03-4200-adba-b2da577b27f3" (UID: "645da40a-5b03-4200-adba-b2da577b27f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:55.318097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.318052    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/645da40a-5b03-4200-adba-b2da577b27f3-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.354114 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.350672    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:52:55.427212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.420823    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-envoy\") pod \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") "
Nov 03 19:52:55.427212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.420949    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-socket\") pod \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") "
Nov 03 19:52:55.427212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.420987    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-data\") pod \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") "
Nov 03 19:52:55.427212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.421006    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-kube-api-access-fspjf\" (UniqueName: \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-kube-api-access-fspjf\") pod \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") "
Nov 03 19:52:55.427212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.421031    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istiod-ca-cert\") pod \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") "
Nov 03 19:52:55.427212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.421093    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-certs\") pod \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") "
Nov 03 19:52:55.427212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.421159    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-podinfo\") pod \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") "
Nov 03 19:52:55.427212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.421289    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-token\" (UniqueName: \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-token\") pod \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") "
Nov 03 19:52:55.427212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.421338    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-credential-socket\") pod \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\" (UID: \"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04\") "
Nov 03 19:52:55.428327 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.427897    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-credential-socket" pod "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" (UID: "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.431419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.431384    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istiod-ca-cert" pod "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" (UID: "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:55.434470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.434440    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-socket" pod "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" (UID: "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.434989 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.434964    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-data" pod "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" (UID: "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.436212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.435394    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-certs" pod "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" (UID: "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.442540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.439008    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:55.447742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.447687    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-envoy" pod "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" (UID: "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.496739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.496515    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-kube-api-access-fspjf" pod "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" (UID: "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04"). InnerVolumeSpecName "kube-api-access-fspjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:55.496739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.496655    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-token" pod "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" (UID: "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:55.496739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.496711    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-podinfo" pod "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" (UID: "896c41d2-a2bf-44a6-abdd-92c3d2ac0a04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:55.533879 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.531064    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.536789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.536655    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.536789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.536700    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.536789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.536719    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fspjf\" (UniqueName: \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-kube-api-access-fspjf\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.536789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.536736    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.536789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.536750    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.536789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.536763    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.536789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.536777    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.536789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.536790    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.699347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.699310    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:55.731761 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:52:55.731731    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73377b59_5b9c_4cb9_8c69_413b5c215d30.slice/crio-acd069bcbad9c5a6eb803e9147eb479beea95686aecc5d302e1c44f11956d2dc WatchSource:0}: Error finding container acd069bcbad9c5a6eb803e9147eb479beea95686aecc5d302e1c44f11956d2dc: Status 404 returned error can't find the container with id acd069bcbad9c5a6eb803e9147eb479beea95686aecc5d302e1c44f11956d2dc
Nov 03 19:52:55.732641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.732617    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"]
Nov 03 19:52:55.746005 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.745901    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"]
Nov 03 19:52:55.747097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.747071    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"
Nov 03 19:52:55.747257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.747089    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj" event={"ID":"896c41d2-a2bf-44a6-abdd-92c3d2ac0a04","Type":"ContainerDied","Data":"14f03f7f6461fdd3fb8e385a806370ca47ad42a6ed5d20f33817e3080f5ec2e7"}
Nov 03 19:52:55.747257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.747154    4531 scope.go:122] "RemoveContainer" containerID="bb710b5d10d4db0b0711e14186983e0cb5fa2d6a7b089a2bc777ec9487f8f643"
Nov 03 19:52:55.748980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.748959    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb" event={"ID":"645da40a-5b03-4200-adba-b2da577b27f3","Type":"ContainerDied","Data":"5f91e934083845a115bab67e95d4ed82822aea85d75cabb1d5cbf3e74b86b6d3"}
Nov 03 19:52:55.748980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.748971    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"
Nov 03 19:52:55.757571 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:52:55.757546    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01880166_1848_4cae_a23b_8ffc842ed06c.slice/crio-f221b0ffb33e88dc1ea2885d5c71e12e3b1f3252ef603153a193cec13944b82a WatchSource:0}: Error finding container f221b0ffb33e88dc1ea2885d5c71e12e3b1f3252ef603153a193cec13944b82a: Status 404 returned error can't find the container with id f221b0ffb33e88dc1ea2885d5c71e12e3b1f3252ef603153a193cec13944b82a
Nov 03 19:52:55.763992 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.763971    4531 scope.go:122] "RemoveContainer" containerID="492e546580bad772080d5d7e0feb0225f8bdd020d4d014d7fe163122f129dcb1"
Nov 03 19:52:55.770627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.770605    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"]
Nov 03 19:52:55.777101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.777079    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-g8cxb"]
Nov 03 19:52:55.784472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.784448    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"]
Nov 03 19:52:55.788906 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:55.788887    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-5rllj"]
Nov 03 19:52:56.155198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.149993    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:56.762468 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.762432    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq" event={"ID":"01880166-1848-4cae-a23b-8ffc842ed06c","Type":"ContainerStarted","Data":"ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b"}
Nov 03 19:52:56.762957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.762486    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq" event={"ID":"01880166-1848-4cae-a23b-8ffc842ed06c","Type":"ContainerStarted","Data":"f221b0ffb33e88dc1ea2885d5c71e12e3b1f3252ef603153a193cec13944b82a"}
Nov 03 19:52:56.762957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.762529    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 19:52:56.764293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.764269    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r" event={"ID":"73377b59-5b9c-4cb9-8c69-413b5c215d30","Type":"ContainerStarted","Data":"ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f"}
Nov 03 19:52:56.764428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.764322    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 19:52:56.764428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.764347    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r" event={"ID":"73377b59-5b9c-4cb9-8c69-413b5c215d30","Type":"ContainerStarted","Data":"acd069bcbad9c5a6eb803e9147eb479beea95686aecc5d302e1c44f11956d2dc"}
Nov 03 19:52:56.781741 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.781682    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq" podStartSLOduration=2.781665661 podStartE2EDuration="2.781665661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:56.777351151 +0000 UTC m=+1290.047474448" watchObservedRunningTime="2026-11-03 19:52:56.781665661 +0000 UTC m=+1290.051788961"
Nov 03 19:52:56.796669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.796618    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r" podStartSLOduration=3.796605992 podStartE2EDuration="3.796605992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:56.790769939 +0000 UTC m=+1290.060893258" watchObservedRunningTime="2026-11-03 19:52:56.796605992 +0000 UTC m=+1290.066729286"
Nov 03 19:52:56.862263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.862232    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645da40a-5b03-4200-adba-b2da577b27f3" path="/var/lib/kubelet/pods/645da40a-5b03-4200-adba-b2da577b27f3/volumes"
Nov 03 19:52:56.862948 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.862930    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" path="/var/lib/kubelet/pods/896c41d2-a2bf-44a6-abdd-92c3d2ac0a04/volumes"
Nov 03 19:52:56.964953 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:56.964903    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:57.531970 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:57.531925    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main" containerID="cri-o://59fd59aa8f96ec15c4b7eee9d5cfd3eb3dbde5d8bdf436eff5a4c789b913db0a" gracePeriod=45
Nov 03 19:52:57.623221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:57.615190    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.160:8000/ready\": dial tcp 10.133.0.160:8000: connect: connection refused"
Nov 03 19:52:57.653061 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:57.653009    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:57.786945 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:57.786624    4531 generic.go:356] "Generic (PLEG): container finished" podID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerID="59fd59aa8f96ec15c4b7eee9d5cfd3eb3dbde5d8bdf436eff5a4c789b913db0a" exitCode=0
Nov 03 19:52:57.787875 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:57.787839    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" event={"ID":"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0","Type":"ContainerDied","Data":"59fd59aa8f96ec15c4b7eee9d5cfd3eb3dbde5d8bdf436eff5a4c789b913db0a"}
Nov 03 19:52:57.908423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:57.908389    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:52:58.114250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.113031    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tmp-dir\") pod \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") "
Nov 03 19:52:58.114250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.113134    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-dshm\") pod \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") "
Nov 03 19:52:58.114250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.113210    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-kube-api-access-wjzbm\" (UniqueName: \"kubernetes.io/projected/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-kube-api-access-wjzbm\") pod \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") "
Nov 03 19:52:58.114250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.113284    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tls-certs\") pod \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") "
Nov 03 19:52:58.114250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.113396    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-home\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-home\") pod \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") "
Nov 03 19:52:58.114250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.113553    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-model-cache\") pod \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\" (UID: \"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0\") "
Nov 03 19:52:58.120194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.114999    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-model-cache" pod "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" (UID: "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:58.120194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.115305    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tmp-dir" pod "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" (UID: "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:58.120194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.116874    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-home" pod "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" (UID: "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:58.205725 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.193533    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-dshm" pod "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" (UID: "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:58.234384 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.234347    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tls-certs" pod "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" (UID: "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:58.234565 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.234526    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-kube-api-access-wjzbm" pod "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" (UID: "825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0"). InnerVolumeSpecName "kube-api-access-wjzbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:58.234731 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.234713    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:58.234829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.234733    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:58.234829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.234742    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:58.234829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.234751    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjzbm\" (UniqueName: \"kubernetes.io/projected/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-kube-api-access-wjzbm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:58.234829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.234762    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:58.234829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.234770    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:58.665197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.661746    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:58.797520 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.797489    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"
Nov 03 19:52:58.798053 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.797526    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl" event={"ID":"825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0","Type":"ContainerDied","Data":"4ab56d188bb18d67096727aa8194c471df32fa8a73976444fba9688af3af22f0"}
Nov 03 19:52:58.798053 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.797580    4531 scope.go:122] "RemoveContainer" containerID="59fd59aa8f96ec15c4b7eee9d5cfd3eb3dbde5d8bdf436eff5a4c789b913db0a"
Nov 03 19:52:58.822397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.822372    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"]
Nov 03 19:52:58.826698 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.826676    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a536f35b/e2e-lifecycle-model-a536f35b-kserve-669578f97c-9ttfl"]
Nov 03 19:52:58.863631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:58.863601    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" path="/var/lib/kubelet/pods/825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0/volumes"
Nov 03 19:52:59.456198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:59.451992    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg" podUID="f432b8b2-d219-4deb-ba65-4604103a176e" containerName="payload-pre-processing" containerID="cri-o://3c62da78e9bc3e5f41277f516a8ddcadfffbccfb79573bc22341442ff15d9b9c" gracePeriod=25
Nov 03 19:52:59.813277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:59.813240    4531 generic.go:356] "Generic (PLEG): container finished" podID="f432b8b2-d219-4deb-ba65-4604103a176e" containerID="3c62da78e9bc3e5f41277f516a8ddcadfffbccfb79573bc22341442ff15d9b9c" exitCode=0
Nov 03 19:52:59.813809 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:59.813313    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg" event={"ID":"f432b8b2-d219-4deb-ba65-4604103a176e","Type":"ContainerDied","Data":"3c62da78e9bc3e5f41277f516a8ddcadfffbccfb79573bc22341442ff15d9b9c"}
Nov 03 19:52:59.813809 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:59.813383    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg" event={"ID":"f432b8b2-d219-4deb-ba65-4604103a176e","Type":"ContainerDied","Data":"ef2b5baf393686257d26f9f9de33397341eed8597768d8ac98f57c47344f9467"}
Nov 03 19:52:59.813809 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:59.813413    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef2b5baf393686257d26f9f9de33397341eed8597768d8ac98f57c47344f9467"
Nov 03 19:52:59.827584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:59.827558    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:52:59.867112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:59.861596    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:59.992107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:59.992021    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f432b8b2-d219-4deb-ba65-4604103a176e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f432b8b2-d219-4deb-ba65-4604103a176e-plugins-config-volume\") pod \"f432b8b2-d219-4deb-ba65-4604103a176e\" (UID: \"f432b8b2-d219-4deb-ba65-4604103a176e\") "
Nov 03 19:52:59.992305 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:59.992137    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f432b8b2-d219-4deb-ba65-4604103a176e-kube-api-access-65g2f\" (UniqueName: \"kubernetes.io/projected/f432b8b2-d219-4deb-ba65-4604103a176e-kube-api-access-65g2f\") pod \"f432b8b2-d219-4deb-ba65-4604103a176e\" (UID: \"f432b8b2-d219-4deb-ba65-4604103a176e\") "
Nov 03 19:52:59.994364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:59.994334    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f432b8b2-d219-4deb-ba65-4604103a176e-plugins-config-volume" pod "f432b8b2-d219-4deb-ba65-4604103a176e" (UID: "f432b8b2-d219-4deb-ba65-4604103a176e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:59.996059 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:52:59.996037    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f432b8b2-d219-4deb-ba65-4604103a176e-kube-api-access-65g2f" pod "f432b8b2-d219-4deb-ba65-4604103a176e" (UID: "f432b8b2-d219-4deb-ba65-4604103a176e"). InnerVolumeSpecName "kube-api-access-65g2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:00.009649 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.009615    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:00.095767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.095735    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f432b8b2-d219-4deb-ba65-4604103a176e-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:00.095767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.095764    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65g2f\" (UniqueName: \"kubernetes.io/projected/f432b8b2-d219-4deb-ba65-4604103a176e-kube-api-access-65g2f\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:00.321254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.321149    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"]
Nov 03 19:53:00.322427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322401    4531 cpu_manager.go:414] "containerMap: removing container" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main"
Nov 03 19:53:00.322427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322425    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main"
Nov 03 19:53:00.322631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322487    4531 cpu_manager.go:414] "containerMap: removing container" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy"
Nov 03 19:53:00.322631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322496    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy"
Nov 03 19:53:00.322631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322509    4531 cpu_manager.go:414] "containerMap: removing container" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerName="istio-proxy"
Nov 03 19:53:00.322631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322517    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerName="istio-proxy"
Nov 03 19:53:00.322631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322539    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f432b8b2-d219-4deb-ba65-4604103a176e" containerName="payload-pre-processing"
Nov 03 19:53:00.322631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322547    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f432b8b2-d219-4deb-ba65-4604103a176e" containerName="payload-pre-processing"
Nov 03 19:53:00.322938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322712    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f432b8b2-d219-4deb-ba65-4604103a176e" containerName="payload-pre-processing"
Nov 03 19:53:00.322938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322732    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="825ddb4c-8e2d-43dd-aa3b-e59f3a5586d0" containerName="main"
Nov 03 19:53:00.322938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322751    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="645da40a-5b03-4200-adba-b2da577b27f3" containerName="istio-proxy"
Nov 03 19:53:00.322938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.322762    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="896c41d2-a2bf-44a6-abdd-92c3d2ac0a04" containerName="istio-proxy"
Nov 03 19:53:00.328179 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.328155    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.336425 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.333239    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"]
Nov 03 19:53:00.336425 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.334572    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ffe132-openshift-default-dockercfg-dp9sn\""
Nov 03 19:53:00.411562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.404365    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-token\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.411562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.404419    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-podinfo\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.411562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.404502    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-data\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.411562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.404527    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505a4164-fe42-4574-aeb6-fbe39f93e798-istiod-ca-cert\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.411562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.404570    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.411562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.404624    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc2dc\" (UniqueName: \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-kube-api-access-fc2dc\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.411562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.404764    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-envoy\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.411562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.404919    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-certs\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.411562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.404954    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-credential-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.418904 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.418869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:00.530196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.523302    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-token\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.530196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.523368    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-podinfo\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.530196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.523458    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-data\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.530196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.523484    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505a4164-fe42-4574-aeb6-fbe39f93e798-istiod-ca-cert\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.530196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.523530    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.530196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.523589    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc2dc\" (UniqueName: \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-kube-api-access-fc2dc\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.530196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.523739    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-envoy\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.530196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.523908    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-certs\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.530196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.523944    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-credential-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.530196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.524602    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-credential-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.549193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.532548    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-data\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.549193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.533283    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505a4164-fe42-4574-aeb6-fbe39f93e798-istiod-ca-cert\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.549193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.533533    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.549193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.543523    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-certs\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.565557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.555958    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc2dc\" (UniqueName: \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-kube-api-access-fc2dc\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.565557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.562803    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:00.565557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.565493    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-envoy\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.573245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.566730    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-token\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.585227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.583093    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-podinfo\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.655202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.654488    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:00.775777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:00.775695    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:01.087194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:01.086626    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"
Nov 03 19:53:01.127945 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:01.127911    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"]
Nov 03 19:53:01.137805 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:01.137229    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b73c810-577674b7ff-5jxgg"]
Nov 03 19:53:01.237072 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:01.237042    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"]
Nov 03 19:53:01.237664 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:01.237640    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505a4164_fe42_4574_aeb6_fbe39f93e798.slice/crio-3bb4e7040293956c1f8fbc3a7280919d3f1d98cb1d527e65bf3fb5f9c81be6e2 WatchSource:0}: Error finding container 3bb4e7040293956c1f8fbc3a7280919d3f1d98cb1d527e65bf3fb5f9c81be6e2: Status 404 returned error can't find the container with id 3bb4e7040293956c1f8fbc3a7280919d3f1d98cb1d527e65bf3fb5f9c81be6e2
Nov 03 19:53:01.241479 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:01.241440    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:01.241557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:01.241527    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:01.241604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:01.241567    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:01.566258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:01.557750    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:02.096221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:02.096185    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" event={"ID":"505a4164-fe42-4574-aeb6-fbe39f93e798","Type":"ContainerStarted","Data":"3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee"}
Nov 03 19:53:02.096598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:02.096241    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" event={"ID":"505a4164-fe42-4574-aeb6-fbe39f93e798","Type":"ContainerStarted","Data":"3bb4e7040293956c1f8fbc3a7280919d3f1d98cb1d527e65bf3fb5f9c81be6e2"}
Nov 03 19:53:02.118934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:02.118879    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" podStartSLOduration=2.118864083 podStartE2EDuration="2.118864083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:02.113600257 +0000 UTC m=+1295.383723578" watchObservedRunningTime="2026-11-03 19:53:02.118864083 +0000 UTC m=+1295.388987377"
Nov 03 19:53:02.322202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:02.317085    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:02.655222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:02.655143    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:02.662184 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:02.662145    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:02.863107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:02.863072    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f432b8b2-d219-4deb-ba65-4604103a176e" path="/var/lib/kubelet/pods/f432b8b2-d219-4deb-ba65-4604103a176e/volumes"
Nov 03 19:53:03.224400 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.224356    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:03.233135 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.233106    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:03.611244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.604290    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"]
Nov 03 19:53:03.611244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.604548    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy" containerID="cri-o://9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708" gracePeriod=30
Nov 03 19:53:03.692747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.685825    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:03.769738 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.769702    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"]
Nov 03 19:53:03.769924 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.769906    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy" containerID="cri-o://9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3" gracePeriod=30
Nov 03 19:53:03.775103 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.775075    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"]
Nov 03 19:53:03.775303 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.775281    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" podUID="02097b90-294d-4de0-bb01-c4bb50228769" containerName="istio-proxy" containerID="cri-o://50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b" gracePeriod=30
Nov 03 19:53:03.951567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.951525    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"]
Nov 03 19:53:03.951758 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.951735    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy" containerID="cri-o://4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff" gracePeriod=30
Nov 03 19:53:03.957991 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.957955    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"]
Nov 03 19:53:03.958302 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.958275    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy" containerID="cri-o://748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9" gracePeriod=30
Nov 03 19:53:03.965077 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.964811    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"]
Nov 03 19:53:03.965077 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:03.965041    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy" containerID="cri-o://36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a" gracePeriod=30
Nov 03 19:53:04.504071 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:04.504035    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:04.643947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:04.643909    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:04.756046 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:04.752245    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 19:53:04.782146 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:04.782106    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:04.782387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:04.782355    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 19:53:05.611840 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.611796    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:05.875877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.875802    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"]
Nov 03 19:53:05.881914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.881891    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 19:53:05.885274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.885237    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ffe132\""
Nov 03 19:53:05.888718 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.888692    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:05.891540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.891520    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"]
Nov 03 19:53:05.917249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.917210    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"]
Nov 03 19:53:05.923763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.923735    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 19:53:05.932755 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.932730    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ffe132-dockercfg-8gmr6\""
Nov 03 19:53:05.940815 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.940774    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"]
Nov 03 19:53:05.993056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.993013    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8blgd\" (UniqueName: \"kubernetes.io/projected/318e30af-1cb1-4dff-8f0a-bdd066e0078b-kube-api-access-8blgd\") pod \"payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp\" (UID: \"318e30af-1cb1-4dff-8f0a-bdd066e0078b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 19:53:05.996739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:05.996667    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318e30af-1cb1-4dff-8f0a-bdd066e0078b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp\" (UID: \"318e30af-1cb1-4dff-8f0a-bdd066e0078b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 19:53:06.060369 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.060329    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:06.144534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.111191    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c45340-7f63-4f26-8eeb-9efc99a62060-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp\" (UID: \"30c45340-7f63-4f26-8eeb-9efc99a62060\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 19:53:06.206717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.206681    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx554\" (UniqueName: \"kubernetes.io/projected/30c45340-7f63-4f26-8eeb-9efc99a62060-kube-api-access-wx554\") pod \"payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp\" (UID: \"30c45340-7f63-4f26-8eeb-9efc99a62060\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 19:53:06.206985 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.206956    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8blgd\" (UniqueName: \"kubernetes.io/projected/318e30af-1cb1-4dff-8f0a-bdd066e0078b-kube-api-access-8blgd\") pod \"payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp\" (UID: \"318e30af-1cb1-4dff-8f0a-bdd066e0078b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 19:53:06.225535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.225491    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318e30af-1cb1-4dff-8f0a-bdd066e0078b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp\" (UID: \"318e30af-1cb1-4dff-8f0a-bdd066e0078b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 19:53:06.367282 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.354004    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c45340-7f63-4f26-8eeb-9efc99a62060-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp\" (UID: \"30c45340-7f63-4f26-8eeb-9efc99a62060\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 19:53:06.427127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.427042    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx554\" (UniqueName: \"kubernetes.io/projected/30c45340-7f63-4f26-8eeb-9efc99a62060-kube-api-access-wx554\") pod \"payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp\" (UID: \"30c45340-7f63-4f26-8eeb-9efc99a62060\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 19:53:06.447192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.382265    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318e30af-1cb1-4dff-8f0a-bdd066e0078b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp\" (UID: \"318e30af-1cb1-4dff-8f0a-bdd066e0078b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 19:53:06.447549 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.424012    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8blgd\" (UniqueName: \"kubernetes.io/projected/318e30af-1cb1-4dff-8f0a-bdd066e0078b-kube-api-access-8blgd\") pod \"payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp\" (UID: \"318e30af-1cb1-4dff-8f0a-bdd066e0078b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 19:53:06.447795 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.354983    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c45340-7f63-4f26-8eeb-9efc99a62060-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp\" (UID: \"30c45340-7f63-4f26-8eeb-9efc99a62060\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 19:53:06.459227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.457096    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx554\" (UniqueName: \"kubernetes.io/projected/30c45340-7f63-4f26-8eeb-9efc99a62060-kube-api-access-wx554\") pod \"payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp\" (UID: \"30c45340-7f63-4f26-8eeb-9efc99a62060\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 19:53:06.500006 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.499967    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 19:53:06.552160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.552123    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 19:53:06.719827 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.719789    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:06.808311 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.808276    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"]
Nov 03 19:53:06.813466 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:06.813431    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318e30af_1cb1_4dff_8f0a_bdd066e0078b.slice/crio-e96617d7b6e21c174b3e4d07b528eec1eeefec4004a4da07f2958a9de6c2955d WatchSource:0}: Error finding container e96617d7b6e21c174b3e4d07b528eec1eeefec4004a4da07f2958a9de6c2955d: Status 404 returned error can't find the container with id e96617d7b6e21c174b3e4d07b528eec1eeefec4004a4da07f2958a9de6c2955d
Nov 03 19:53:06.917988 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:06.917958    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c45340_7f63_4f26_8eeb_9efc99a62060.slice/crio-ffb69eb05d7fe255e84993de0512f0768b7b49efaddf2c8e12275b40ca4591b7 WatchSource:0}: Error finding container ffb69eb05d7fe255e84993de0512f0768b7b49efaddf2c8e12275b40ca4591b7: Status 404 returned error can't find the container with id ffb69eb05d7fe255e84993de0512f0768b7b49efaddf2c8e12275b40ca4591b7
Nov 03 19:53:06.918390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:06.918365    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"]
Nov 03 19:53:07.388845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.388801    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp" event={"ID":"30c45340-7f63-4f26-8eeb-9efc99a62060","Type":"ContainerStarted","Data":"c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69"}
Nov 03 19:53:07.389018 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.388885    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp" event={"ID":"30c45340-7f63-4f26-8eeb-9efc99a62060","Type":"ContainerStarted","Data":"ffb69eb05d7fe255e84993de0512f0768b7b49efaddf2c8e12275b40ca4591b7"}
Nov 03 19:53:07.389131 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.389054    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 19:53:07.391127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.391092    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp" event={"ID":"318e30af-1cb1-4dff-8f0a-bdd066e0078b","Type":"ContainerStarted","Data":"7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3"}
Nov 03 19:53:07.391321 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.391147    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp" event={"ID":"318e30af-1cb1-4dff-8f0a-bdd066e0078b","Type":"ContainerStarted","Data":"e96617d7b6e21c174b3e4d07b528eec1eeefec4004a4da07f2958a9de6c2955d"}
Nov 03 19:53:07.391760 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.391736    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 19:53:07.406587 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.406443    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp" podStartSLOduration=2.406425618 podStartE2EDuration="2.406425618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:07.402331768 +0000 UTC m=+1300.672455065" watchObservedRunningTime="2026-11-03 19:53:07.406425618 +0000 UTC m=+1300.676548916"
Nov 03 19:53:07.407922 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.407718    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:07.419412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.419362    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp" podStartSLOduration=2.419349596 podStartE2EDuration="2.419349596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:07.417464067 +0000 UTC m=+1300.687587362" watchObservedRunningTime="2026-11-03 19:53:07.419349596 +0000 UTC m=+1300.689472891"
Nov 03 19:53:07.554200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.546765    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:07.791128 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.791026    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 19:53:07.791667 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.791492    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 19:53:07.960831 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:07.960788    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:08.409410 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:08.399437    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:09.015788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.015725    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"]
Nov 03 19:53:09.023102 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.023082    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:09.024681 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.024653    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"]
Nov 03 19:53:09.025781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.025760    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:53:09.025910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.025896    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:53:09.054999 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.054972    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sfxr\" (UniqueName: \"kubernetes.io/projected/0d45979f-2173-4f23-ba2d-9ffa2f40e087-kube-api-access-2sfxr\") pod \"maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:09.055211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.055190    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:09.055352 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.055333    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:09.166257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.158687    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sfxr\" (UniqueName: \"kubernetes.io/projected/0d45979f-2173-4f23-ba2d-9ffa2f40e087-kube-api-access-2sfxr\") pod \"maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:09.166257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.158740    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:09.166257 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.158777    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:09.188110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.176885    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:09.214601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.213831    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:53:09.244823 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.244675    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.270765    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-credential-socket\") pod \"5a353656-7621-47f3-a298-ef89b4a6ced9\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.270785    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sfxr\" (UniqueName: \"kubernetes.io/projected/0d45979f-2173-4f23-ba2d-9ffa2f40e087-kube-api-access-2sfxr\") pod \"maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.270816    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-kube-api-access-wdbhr\" (UniqueName: \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-kube-api-access-wdbhr\") pod \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.270862    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-socket\") pod \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.270952    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-podinfo\") pod \"5a353656-7621-47f3-a298-ef89b4a6ced9\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.270993    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istiod-ca-cert\") pod \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271035    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-credential-socket\") pod \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271039    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-credential-socket" pod "5a353656-7621-47f3-a298-ef89b4a6ced9" (UID: "5a353656-7621-47f3-a298-ef89b4a6ced9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271065    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-data\") pod \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271102    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a353656-7621-47f3-a298-ef89b4a6ced9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a353656-7621-47f3-a298-ef89b4a6ced9-istiod-ca-cert\") pod \"5a353656-7621-47f3-a298-ef89b4a6ced9\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271138    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-token\" (UniqueName: \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-token\") pod \"5a353656-7621-47f3-a298-ef89b4a6ced9\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271190    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-token\" (UniqueName: \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-token\") pod \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271228    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-data\") pod \"5a353656-7621-47f3-a298-ef89b4a6ced9\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271260    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-socket\") pod \"5a353656-7621-47f3-a298-ef89b4a6ced9\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271328    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-kube-api-access-s2rms\" (UniqueName: \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-kube-api-access-s2rms\") pod \"5a353656-7621-47f3-a298-ef89b4a6ced9\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271386    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-podinfo\") pod \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271522    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-envoy\") pod \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271627    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-certs\") pod \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\" (UID: \"23301cd8-4fc8-4544-9744-4fa7cf53ff7c\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271658    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-envoy\") pod \"5a353656-7621-47f3-a298-ef89b4a6ced9\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.271966    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-certs\") pod \"5a353656-7621-47f3-a298-ef89b4a6ced9\" (UID: \"5a353656-7621-47f3-a298-ef89b4a6ced9\") "
Nov 03 19:53:09.273208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.272959    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.274480 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.273853    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-podinfo" pod "5a353656-7621-47f3-a298-ef89b4a6ced9" (UID: "5a353656-7621-47f3-a298-ef89b4a6ced9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:09.275226 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.274972    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istiod-ca-cert" pod "23301cd8-4fc8-4544-9744-4fa7cf53ff7c" (UID: "23301cd8-4fc8-4544-9744-4fa7cf53ff7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:09.276782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.275448    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-credential-socket" pod "23301cd8-4fc8-4544-9744-4fa7cf53ff7c" (UID: "23301cd8-4fc8-4544-9744-4fa7cf53ff7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.276782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.276593    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-kube-api-access-wdbhr" pod "23301cd8-4fc8-4544-9744-4fa7cf53ff7c" (UID: "23301cd8-4fc8-4544-9744-4fa7cf53ff7c"). InnerVolumeSpecName "kube-api-access-wdbhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.276962 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.276785    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-socket" pod "23301cd8-4fc8-4544-9744-4fa7cf53ff7c" (UID: "23301cd8-4fc8-4544-9744-4fa7cf53ff7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.277440 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.277085    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-data" pod "23301cd8-4fc8-4544-9744-4fa7cf53ff7c" (UID: "23301cd8-4fc8-4544-9744-4fa7cf53ff7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.280621 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.279987    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-kube-api-access-s2rms" pod "5a353656-7621-47f3-a298-ef89b4a6ced9" (UID: "5a353656-7621-47f3-a298-ef89b4a6ced9"). InnerVolumeSpecName "kube-api-access-s2rms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.280621 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.280346    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a353656-7621-47f3-a298-ef89b4a6ced9-istiod-ca-cert" pod "5a353656-7621-47f3-a298-ef89b4a6ced9" (UID: "5a353656-7621-47f3-a298-ef89b4a6ced9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:09.285857 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.285661    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-token" pod "5a353656-7621-47f3-a298-ef89b4a6ced9" (UID: "5a353656-7621-47f3-a298-ef89b4a6ced9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.289955 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.289930    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-data" pod "5a353656-7621-47f3-a298-ef89b4a6ced9" (UID: "5a353656-7621-47f3-a298-ef89b4a6ced9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.290361 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.290341    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-socket" pod "5a353656-7621-47f3-a298-ef89b4a6ced9" (UID: "5a353656-7621-47f3-a298-ef89b4a6ced9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.291137 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.291102    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-token" pod "23301cd8-4fc8-4544-9744-4fa7cf53ff7c" (UID: "23301cd8-4fc8-4544-9744-4fa7cf53ff7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.291285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.291138    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-certs" pod "5a353656-7621-47f3-a298-ef89b4a6ced9" (UID: "5a353656-7621-47f3-a298-ef89b4a6ced9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.291438 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.291417    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-certs" pod "23301cd8-4fc8-4544-9744-4fa7cf53ff7c" (UID: "23301cd8-4fc8-4544-9744-4fa7cf53ff7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.299109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.299068    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-envoy" pod "5a353656-7621-47f3-a298-ef89b4a6ced9" (UID: "5a353656-7621-47f3-a298-ef89b4a6ced9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.302091 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.302069    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-envoy" pod "23301cd8-4fc8-4544-9744-4fa7cf53ff7c" (UID: "23301cd8-4fc8-4544-9744-4fa7cf53ff7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.302414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.302390    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:09.304666 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.304640    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-podinfo" pod "23301cd8-4fc8-4544-9744-4fa7cf53ff7c" (UID: "23301cd8-4fc8-4544-9744-4fa7cf53ff7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:09.340518 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.340091    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:53:09.374008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.373975    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-certs\") pod \"02097b90-294d-4de0-bb01-c4bb50228769\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") "
Nov 03 19:53:09.374413 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.374387    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-socket\") pod \"02097b90-294d-4de0-bb01-c4bb50228769\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") "
Nov 03 19:53:09.374742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.374721    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-kube-api-access-d5j2b\" (UniqueName: \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-kube-api-access-d5j2b\") pod \"02097b90-294d-4de0-bb01-c4bb50228769\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") "
Nov 03 19:53:09.374947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.374930    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-credential-socket\") pod \"02097b90-294d-4de0-bb01-c4bb50228769\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") "
Nov 03 19:53:09.375132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.375114    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-data\") pod \"02097b90-294d-4de0-bb01-c4bb50228769\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") "
Nov 03 19:53:09.375357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.375340    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02097b90-294d-4de0-bb01-c4bb50228769-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02097b90-294d-4de0-bb01-c4bb50228769-istio-podinfo\") pod \"02097b90-294d-4de0-bb01-c4bb50228769\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") "
Nov 03 19:53:09.375508 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.375493    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-envoy\") pod \"02097b90-294d-4de0-bb01-c4bb50228769\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") "
Nov 03 19:53:09.375918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.375896    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02097b90-294d-4de0-bb01-c4bb50228769-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02097b90-294d-4de0-bb01-c4bb50228769-istiod-ca-cert\") pod \"02097b90-294d-4de0-bb01-c4bb50228769\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") "
Nov 03 19:53:09.376158 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.376137    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-istio-token\" (UniqueName: \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-istio-token\") pod \"02097b90-294d-4de0-bb01-c4bb50228769\" (UID: \"02097b90-294d-4de0-bb01-c4bb50228769\") "
Nov 03 19:53:09.377641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.377620    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.378259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.378238    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdbhr\" (UniqueName: \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-kube-api-access-wdbhr\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.378377 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.378364    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.378465 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.378453    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.378548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.378536    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.378658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.378644    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.378751 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.378739    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.378828 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.378816    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a353656-7621-47f3-a298-ef89b4a6ced9-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.378906 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.378895    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.378990 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.378978    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.379078 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.379068    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.379158 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.379147    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.379254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.379241    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2rms\" (UniqueName: \"kubernetes.io/projected/5a353656-7621-47f3-a298-ef89b4a6ced9-kube-api-access-s2rms\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.379330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.379318    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.379398 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.379389    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.379479 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.379468    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23301cd8-4fc8-4544-9744-4fa7cf53ff7c-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.379590 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.379576    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a353656-7621-47f3-a298-ef89b4a6ced9-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.382244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.374503    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-certs" pod "02097b90-294d-4de0-bb01-c4bb50228769" (UID: "02097b90-294d-4de0-bb01-c4bb50228769"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.382244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.374753    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-socket" pod "02097b90-294d-4de0-bb01-c4bb50228769" (UID: "02097b90-294d-4de0-bb01-c4bb50228769"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.382244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.380376    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-data" pod "02097b90-294d-4de0-bb01-c4bb50228769" (UID: "02097b90-294d-4de0-bb01-c4bb50228769"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.382244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.380777    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-credential-socket" pod "02097b90-294d-4de0-bb01-c4bb50228769" (UID: "02097b90-294d-4de0-bb01-c4bb50228769"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.382244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.381452    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02097b90-294d-4de0-bb01-c4bb50228769-istiod-ca-cert" pod "02097b90-294d-4de0-bb01-c4bb50228769" (UID: "02097b90-294d-4de0-bb01-c4bb50228769"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:09.396096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.395819    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-istio-token" pod "02097b90-294d-4de0-bb01-c4bb50228769" (UID: "02097b90-294d-4de0-bb01-c4bb50228769"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.409871 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.409474    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:53:09.418406 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.418377    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02097b90-294d-4de0-bb01-c4bb50228769-istio-podinfo" pod "02097b90-294d-4de0-bb01-c4bb50228769" (UID: "02097b90-294d-4de0-bb01-c4bb50228769"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:09.426852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.426804    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-envoy" pod "02097b90-294d-4de0-bb01-c4bb50228769" (UID: "02097b90-294d-4de0-bb01-c4bb50228769"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.428723 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.428684    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-kube-api-access-d5j2b" pod "02097b90-294d-4de0-bb01-c4bb50228769" (UID: "02097b90-294d-4de0-bb01-c4bb50228769"). InnerVolumeSpecName "kube-api-access-d5j2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.430977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.430943    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"]
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433098    4531 cpu_manager.go:414] "containerMap: removing container" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy"
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433383    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy"
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433438    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy"
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433447    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy"
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433617    4531 cpu_manager.go:414] "containerMap: removing container" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy"
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433630    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy"
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433660    4531 cpu_manager.go:414] "containerMap: removing container" podUID="02097b90-294d-4de0-bb01-c4bb50228769" containerName="istio-proxy"
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433668    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02097b90-294d-4de0-bb01-c4bb50228769" containerName="istio-proxy"
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433879    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="04530f22-b013-40fe-8390-81a0ea009267" containerName="istio-proxy"
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433906    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerName="istio-proxy"
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433933    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerName="istio-proxy"
Nov 03 19:53:09.434804 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.433948    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="02097b90-294d-4de0-bb01-c4bb50228769" containerName="istio-proxy"
Nov 03 19:53:09.462704 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.441496    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.467029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.465678    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-9f3188-serving-cert\""
Nov 03 19:53:09.481948 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.481912    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04530f22-b013-40fe-8390-81a0ea009267-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04530f22-b013-40fe-8390-81a0ea009267-istiod-ca-cert\") pod \"04530f22-b013-40fe-8390-81a0ea009267\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") "
Nov 03 19:53:09.482153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.482030    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04530f22-b013-40fe-8390-81a0ea009267-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04530f22-b013-40fe-8390-81a0ea009267-istio-podinfo\") pod \"04530f22-b013-40fe-8390-81a0ea009267\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") "
Nov 03 19:53:09.482153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.482132    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-certs\") pod \"04530f22-b013-40fe-8390-81a0ea009267\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") "
Nov 03 19:53:09.482353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.482198    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-socket\") pod \"04530f22-b013-40fe-8390-81a0ea009267\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") "
Nov 03 19:53:09.482353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.482280    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-envoy\") pod \"04530f22-b013-40fe-8390-81a0ea009267\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") "
Nov 03 19:53:09.482466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.482385    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-credential-socket\") pod \"04530f22-b013-40fe-8390-81a0ea009267\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") "
Nov 03 19:53:09.482466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.482415    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-data\") pod \"04530f22-b013-40fe-8390-81a0ea009267\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") "
Nov 03 19:53:09.482466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.482444    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-kube-api-access-jwnt4\" (UniqueName: \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-kube-api-access-jwnt4\") pod \"04530f22-b013-40fe-8390-81a0ea009267\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") "
Nov 03 19:53:09.482631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.482560    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-istio-token\" (UniqueName: \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-istio-token\") pod \"04530f22-b013-40fe-8390-81a0ea009267\" (UID: \"04530f22-b013-40fe-8390-81a0ea009267\") "
Nov 03 19:53:09.482960 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.482938    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483095    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483217    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e841f9-77b1-4f9e-bf90-a444d25f01d7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483502    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrvd8\" (UniqueName: \"kubernetes.io/projected/90e841f9-77b1-4f9e-bf90-a444d25f01d7-kube-api-access-xrvd8\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483608    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483624    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483640    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5j2b\" (UniqueName: \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-kube-api-access-d5j2b\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483656    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483669    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483683    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02097b90-294d-4de0-bb01-c4bb50228769-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483696    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02097b90-294d-4de0-bb01-c4bb50228769-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483709    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02097b90-294d-4de0-bb01-c4bb50228769-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.484286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.483724    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02097b90-294d-4de0-bb01-c4bb50228769-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.487868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.487824    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-data" pod "04530f22-b013-40fe-8390-81a0ea009267" (UID: "04530f22-b013-40fe-8390-81a0ea009267"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.489418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.489387    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04530f22-b013-40fe-8390-81a0ea009267-istiod-ca-cert" pod "04530f22-b013-40fe-8390-81a0ea009267" (UID: "04530f22-b013-40fe-8390-81a0ea009267"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:09.489624 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.489553    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-certs" pod "04530f22-b013-40fe-8390-81a0ea009267" (UID: "04530f22-b013-40fe-8390-81a0ea009267"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.490018 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.489770    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-socket" pod "04530f22-b013-40fe-8390-81a0ea009267" (UID: "04530f22-b013-40fe-8390-81a0ea009267"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.491345 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.491119    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-credential-socket" pod "04530f22-b013-40fe-8390-81a0ea009267" (UID: "04530f22-b013-40fe-8390-81a0ea009267"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.491579 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.491441    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04530f22-b013-40fe-8390-81a0ea009267-istio-podinfo" pod "04530f22-b013-40fe-8390-81a0ea009267" (UID: "04530f22-b013-40fe-8390-81a0ea009267"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:09.495554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.495050    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-istio-token" pod "04530f22-b013-40fe-8390-81a0ea009267" (UID: "04530f22-b013-40fe-8390-81a0ea009267"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.495554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.495353    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-kube-api-access-jwnt4" pod "04530f22-b013-40fe-8390-81a0ea009267" (UID: "04530f22-b013-40fe-8390-81a0ea009267"). InnerVolumeSpecName "kube-api-access-jwnt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.498698 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.498665    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-envoy" pod "04530f22-b013-40fe-8390-81a0ea009267" (UID: "04530f22-b013-40fe-8390-81a0ea009267"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.510956 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.510660    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:53:09.518815 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.518755    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:09.530358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.530334    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"]
Nov 03 19:53:09.542255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.541008    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:53:09.553089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.553062    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:09.553763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.553733    4531 generic.go:356] "Generic (PLEG): container finished" podID="02097b90-294d-4de0-bb01-c4bb50228769" containerID="50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b" exitCode=0
Nov 03 19:53:09.553858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.553833    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" event={"ID":"02097b90-294d-4de0-bb01-c4bb50228769","Type":"ContainerDied","Data":"50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b"}
Nov 03 19:53:09.553903 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.553883    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86" event={"ID":"02097b90-294d-4de0-bb01-c4bb50228769","Type":"ContainerDied","Data":"fd84c1e487a3a309c9847cbbb5eeaf571b18da9263ba02479463061ab37f78c2"}
Nov 03 19:53:09.553941 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.553913    4531 scope.go:122] "RemoveContainer" containerID="50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b"
Nov 03 19:53:09.554101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.554084    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"
Nov 03 19:53:09.576446 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.576417    4531 generic.go:356] "Generic (PLEG): container finished" podID="04530f22-b013-40fe-8390-81a0ea009267" containerID="36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a" exitCode=0
Nov 03 19:53:09.577427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.576474    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"
Nov 03 19:53:09.577635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.576547    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" event={"ID":"04530f22-b013-40fe-8390-81a0ea009267","Type":"ContainerDied","Data":"36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a"}
Nov 03 19:53:09.577709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.577688    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt" event={"ID":"04530f22-b013-40fe-8390-81a0ea009267","Type":"ContainerDied","Data":"c527cfdd4ee8d25d7f92a34640cd730f3e1fa1c18cb4a1bbe2fb99ebfac83cf1"}
Nov 03 19:53:09.583568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.583544    4531 generic.go:356] "Generic (PLEG): container finished" podID="5a353656-7621-47f3-a298-ef89b4a6ced9" containerID="9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708" exitCode=0
Nov 03 19:53:09.583833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.583683    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" event={"ID":"5a353656-7621-47f3-a298-ef89b4a6ced9","Type":"ContainerDied","Data":"9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708"}
Nov 03 19:53:09.583933 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.583868    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94" event={"ID":"5a353656-7621-47f3-a298-ef89b4a6ced9","Type":"ContainerDied","Data":"396447d60a723baff08249d4def8735ddcd8e15e4567171e5efc67ab2ff59b63"}
Nov 03 19:53:09.584605 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.583926    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"
Nov 03 19:53:09.585548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.584487    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-podinfo\") pod \"7e1a97c8-d12c-46eb-a635-e450998128d5\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") "
Nov 03 19:53:09.585730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.585713    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-data\") pod \"7e1a97c8-d12c-46eb-a635-e450998128d5\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") "
Nov 03 19:53:09.591962 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.585921    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-credential-socket\") pod \"7e1a97c8-d12c-46eb-a635-e450998128d5\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") "
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.597419    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-credential-socket\") pod \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") "
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.597495    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-envoy\") pod \"7e1a97c8-d12c-46eb-a635-e450998128d5\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") "
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.597552    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istiod-ca-cert\") pod \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") "
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.597589    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-token\" (UniqueName: \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-token\") pod \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") "
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.597678    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-certs\") pod \"7e1a97c8-d12c-46eb-a635-e450998128d5\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") "
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.597747    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-data" pod "7e1a97c8-d12c-46eb-a635-e450998128d5" (UID: "7e1a97c8-d12c-46eb-a635-e450998128d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.597777    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-certs\") pod \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") "
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.597835    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-kube-api-access-9fkqs\" (UniqueName: \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-kube-api-access-9fkqs\") pod \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") "
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.597984    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1a97c8-d12c-46eb-a635-e450998128d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1a97c8-d12c-46eb-a635-e450998128d5-istiod-ca-cert\") pod \"7e1a97c8-d12c-46eb-a635-e450998128d5\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") "
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.598077    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-envoy\") pod \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") "
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.598099    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-credential-socket" pod "7e1a97c8-d12c-46eb-a635-e450998128d5" (UID: "7e1a97c8-d12c-46eb-a635-e450998128d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.598198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.598129    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-socket\") pod \"7e1a97c8-d12c-46eb-a635-e450998128d5\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") "
Nov 03 19:53:09.598952 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.598391    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-credential-socket" pod "bebdbfae-b6b0-4ea1-a180-29de52d1cc60" (UID: "bebdbfae-b6b0-4ea1-a180-29de52d1cc60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.600688 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.598995    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istiod-ca-cert" pod "bebdbfae-b6b0-4ea1-a180-29de52d1cc60" (UID: "bebdbfae-b6b0-4ea1-a180-29de52d1cc60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:09.600688 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.599078    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-data\") pod \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") "
Nov 03 19:53:09.600688 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.599159    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-socket\") pod \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") "
Nov 03 19:53:09.600688 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.599276    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-kube-api-access-gkzbv\" (UniqueName: \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-kube-api-access-gkzbv\") pod \"7e1a97c8-d12c-46eb-a635-e450998128d5\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") "
Nov 03 19:53:09.600688 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.599310    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-certs" pod "bebdbfae-b6b0-4ea1-a180-29de52d1cc60" (UID: "bebdbfae-b6b0-4ea1-a180-29de52d1cc60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.600688 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.599341    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-podinfo\") pod \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\" (UID: \"bebdbfae-b6b0-4ea1-a180-29de52d1cc60\") "
Nov 03 19:53:09.600688 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.599479    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-token\" (UniqueName: \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-token\") pod \"7e1a97c8-d12c-46eb-a635-e450998128d5\" (UID: \"7e1a97c8-d12c-46eb-a635-e450998128d5\") "
Nov 03 19:53:09.600688 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.600633    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrvd8\" (UniqueName: \"kubernetes.io/projected/90e841f9-77b1-4f9e-bf90-a444d25f01d7-kube-api-access-xrvd8\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.601233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.600783    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.601233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.600963    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.601233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601087    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e841f9-77b1-4f9e-bf90-a444d25f01d7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.601396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601370    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601395    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601414    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601420    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-data" pod "bebdbfae-b6b0-4ea1-a180-29de52d1cc60" (UID: "bebdbfae-b6b0-4ea1-a180-29de52d1cc60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.601456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601441    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601459    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601474    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04530f22-b013-40fe-8390-81a0ea009267-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601492    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwnt4\" (UniqueName: \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-kube-api-access-jwnt4\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601513    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04530f22-b013-40fe-8390-81a0ea009267-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601528    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601543    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04530f22-b013-40fe-8390-81a0ea009267-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601558    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601579    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601596    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.601661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.601611    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04530f22-b013-40fe-8390-81a0ea009267-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.602504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.602479    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e841f9-77b1-4f9e-bf90-a444d25f01d7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.602757 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.602734    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-socket" pod "bebdbfae-b6b0-4ea1-a180-29de52d1cc60" (UID: "bebdbfae-b6b0-4ea1-a180-29de52d1cc60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.619030 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.612916    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-certs" pod "7e1a97c8-d12c-46eb-a635-e450998128d5" (UID: "7e1a97c8-d12c-46eb-a635-e450998128d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.619030 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.612937    4531 scope.go:122] "RemoveContainer" containerID="50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b"
Nov 03 19:53:09.619030 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.615621    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-socket" pod "7e1a97c8-d12c-46eb-a635-e450998128d5" (UID: "7e1a97c8-d12c-46eb-a635-e450998128d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.620051 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.619462    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-podinfo" pod "7e1a97c8-d12c-46eb-a635-e450998128d5" (UID: "7e1a97c8-d12c-46eb-a635-e450998128d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:09.620051 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.619615    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-token" pod "bebdbfae-b6b0-4ea1-a180-29de52d1cc60" (UID: "bebdbfae-b6b0-4ea1-a180-29de52d1cc60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.620051 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.619678    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-envoy" pod "7e1a97c8-d12c-46eb-a635-e450998128d5" (UID: "7e1a97c8-d12c-46eb-a635-e450998128d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.627612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.620750    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1a97c8-d12c-46eb-a635-e450998128d5-istiod-ca-cert" pod "7e1a97c8-d12c-46eb-a635-e450998128d5" (UID: "7e1a97c8-d12c-46eb-a635-e450998128d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:09.627612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.621918    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-kube-api-access-9fkqs" pod "bebdbfae-b6b0-4ea1-a180-29de52d1cc60" (UID: "bebdbfae-b6b0-4ea1-a180-29de52d1cc60"). InnerVolumeSpecName "kube-api-access-9fkqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.627612 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.622041    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-kube-api-access-gkzbv" pod "7e1a97c8-d12c-46eb-a635-e450998128d5" (UID: "7e1a97c8-d12c-46eb-a635-e450998128d5"). InnerVolumeSpecName "kube-api-access-gkzbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.645205 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:53:09.631580    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b\": container with ID starting with 50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b not found: ID does not exist" containerID="50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b"
Nov 03 19:53:09.645205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.631625    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b"} err="failed to get container status \"50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b\": rpc error: code = NotFound desc = could not find container \"50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b\": container with ID starting with 50a4dd539ccff9e47d465d1a0b75f47962babaceb95072bb192fbfe604671d9b not found: ID does not exist"
Nov 03 19:53:09.645205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.631656    4531 scope.go:122] "RemoveContainer" containerID="36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a"
Nov 03 19:53:09.645205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.641340    4531 generic.go:356] "Generic (PLEG): container finished" podID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerID="4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff" exitCode=0
Nov 03 19:53:09.645205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.641523    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" event={"ID":"7e1a97c8-d12c-46eb-a635-e450998128d5","Type":"ContainerDied","Data":"4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff"}
Nov 03 19:53:09.645205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.641618    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6" event={"ID":"7e1a97c8-d12c-46eb-a635-e450998128d5","Type":"ContainerDied","Data":"53c9daf1470dde811a1820f85ef5e5ac9c5c4a8876d6b7870e2fff51d7b1676d"}
Nov 03 19:53:09.645205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.641738    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"
Nov 03 19:53:09.646879 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.646741    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrvd8\" (UniqueName: \"kubernetes.io/projected/90e841f9-77b1-4f9e-bf90-a444d25f01d7-kube-api-access-xrvd8\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.651788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.651704    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-token" pod "7e1a97c8-d12c-46eb-a635-e450998128d5" (UID: "7e1a97c8-d12c-46eb-a635-e450998128d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.652354 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.652333    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.666229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.653768    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.667019 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.666991    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-podinfo" pod "bebdbfae-b6b0-4ea1-a180-29de52d1cc60" (UID: "bebdbfae-b6b0-4ea1-a180-29de52d1cc60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:09.672233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.667779    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"]
Nov 03 19:53:09.673668 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.673626    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-envoy" pod "bebdbfae-b6b0-4ea1-a180-29de52d1cc60" (UID: "bebdbfae-b6b0-4ea1-a180-29de52d1cc60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:09.685547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.677658    4531 generic.go:356] "Generic (PLEG): container finished" podID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerID="748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9" exitCode=0
Nov 03 19:53:09.685547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.677954    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" event={"ID":"bebdbfae-b6b0-4ea1-a180-29de52d1cc60","Type":"ContainerDied","Data":"748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9"}
Nov 03 19:53:09.685547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.680780    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t" event={"ID":"bebdbfae-b6b0-4ea1-a180-29de52d1cc60","Type":"ContainerDied","Data":"6a86d7e06788eb686cf70ae9bd44bf4f3af4a3acdb5260544d41972cd9df4b43"}
Nov 03 19:53:09.685547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.680968    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"
Nov 03 19:53:09.685547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.684948    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:09.687349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.685617    4531 scope.go:122] "RemoveContainer" containerID="36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a"
Nov 03 19:53:09.687349 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:53:09.686274    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a\": container with ID starting with 36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a not found: ID does not exist" containerID="36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a"
Nov 03 19:53:09.687349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.686315    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a"} err="failed to get container status \"36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a\": rpc error: code = NotFound desc = could not find container \"36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a\": container with ID starting with 36a34fb07b5556911579f3ec22dcdd5f3b94f09abae27055c159e66cfe220a1a not found: ID does not exist"
Nov 03 19:53:09.687349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.686338    4531 scope.go:122] "RemoveContainer" containerID="9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708"
Nov 03 19:53:09.694630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.691471    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-x7t86"]
Nov 03 19:53:09.720571 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.720485    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"]
Nov 03 19:53:09.907578 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.825016    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:09.911690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.858981    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.911690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.911055    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.911690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.911075    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.911690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.911091    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.911690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.911105    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.911690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.911658    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fkqs\" (UniqueName: \"kubernetes.io/projected/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-kube-api-access-9fkqs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.911690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.911685    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1a97c8-d12c-46eb-a635-e450998128d5-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.912939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.911698    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.912939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.911711    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1a97c8-d12c-46eb-a635-e450998128d5-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.912939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.911724    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.912939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.911737    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.912939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.911751    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkzbv\" (UniqueName: \"kubernetes.io/projected/7e1a97c8-d12c-46eb-a635-e450998128d5-kube-api-access-gkzbv\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.913593 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.910570    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:09.913815 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.913767    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kgk94"]
Nov 03 19:53:09.921845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.913946    4531 generic.go:356] "Generic (PLEG): container finished" podID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" containerID="9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3" exitCode=0
Nov 03 19:53:09.921845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.914832    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"
Nov 03 19:53:09.921845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.915071    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" event={"ID":"23301cd8-4fc8-4544-9744-4fa7cf53ff7c","Type":"ContainerDied","Data":"9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3"}
Nov 03 19:53:09.921845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.915134    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl" event={"ID":"23301cd8-4fc8-4544-9744-4fa7cf53ff7c","Type":"ContainerDied","Data":"2af372d5ebd04110269fb747f24da50a7d666f76b3a2ea73d125b3ffb2942ed2"}
Nov 03 19:53:09.921845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.919137    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"]
Nov 03 19:53:09.921845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.921154    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebdbfae-b6b0-4ea1-a180-29de52d1cc60-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.927695 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.926861    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wptxt"]
Nov 03 19:53:09.934272 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.932907    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"]
Nov 03 19:53:09.940218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.940112    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-hd78t"]
Nov 03 19:53:09.944858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.944834    4531 scope.go:122] "RemoveContainer" containerID="9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708"
Nov 03 19:53:09.945266 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:53:09.945242    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708\": container with ID starting with 9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708 not found: ID does not exist" containerID="9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708"
Nov 03 19:53:09.945370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.945286    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708"} err="failed to get container status \"9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708\": rpc error: code = NotFound desc = could not find container \"9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708\": container with ID starting with 9cf7487b122b602874c07fe3a4f155872c13c2c2f564da9abf4d8d00e06c7708 not found: ID does not exist"
Nov 03 19:53:09.945370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.945304    4531 scope.go:122] "RemoveContainer" containerID="4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff"
Nov 03 19:53:09.960075 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.960055    4531 scope.go:122] "RemoveContainer" containerID="4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff"
Nov 03 19:53:09.960548 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:53:09.960527    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff\": container with ID starting with 4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff not found: ID does not exist" containerID="4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff"
Nov 03 19:53:09.960702 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.960678    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff"} err="failed to get container status \"4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff\": rpc error: code = NotFound desc = could not find container \"4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff\": container with ID starting with 4c26fb6a79f28c9249a1690072b279a88ebc4e7660035326c10376ddd136f5ff not found: ID does not exist"
Nov 03 19:53:09.960799 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.960786    4531 scope.go:122] "RemoveContainer" containerID="748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9"
Nov 03 19:53:09.961136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.960937    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"]
Nov 03 19:53:09.967755 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:09.967727    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-tnbdl"]
Nov 03 19:53:10.003839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.003810    4531 scope.go:122] "RemoveContainer" containerID="748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9"
Nov 03 19:53:10.005357 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:53:10.005306    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9\": container with ID starting with 748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9 not found: ID does not exist" containerID="748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9"
Nov 03 19:53:10.005504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.005479    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9"} err="failed to get container status \"748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9\": rpc error: code = NotFound desc = could not find container \"748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9\": container with ID starting with 748975a71ad2bf45b89fd5deb02a303497dba6f1d6637975e3df338c8cdc19f9 not found: ID does not exist"
Nov 03 19:53:10.005608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.005596    4531 scope.go:122] "RemoveContainer" containerID="9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3"
Nov 03 19:53:10.024338 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.019313    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"]
Nov 03 19:53:10.028617 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.028368    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-wf6b6"]
Nov 03 19:53:10.028795 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.028612    4531 scope.go:122] "RemoveContainer" containerID="9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3"
Nov 03 19:53:10.029307 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:53:10.029231    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3\": container with ID starting with 9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3 not found: ID does not exist" containerID="9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3"
Nov 03 19:53:10.029307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.029268    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3"} err="failed to get container status \"9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3\": rpc error: code = NotFound desc = could not find container \"9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3\": container with ID starting with 9e98476947f2629e5d873274492928ce6ed03e27d3aaa16bb412c27bfe07bfb3 not found: ID does not exist"
Nov 03 19:53:10.106245 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:10.106214    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d45979f_2173_4f23_ba2d_9ffa2f40e087.slice/crio-84727648a2a90e70c637e26bc3cd564f8bec95e31f904999c077986571f12bb7 WatchSource:0}: Error finding container 84727648a2a90e70c637e26bc3cd564f8bec95e31f904999c077986571f12bb7: Status 404 returned error can't find the container with id 84727648a2a90e70c637e26bc3cd564f8bec95e31f904999c077986571f12bb7
Nov 03 19:53:10.106906 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.106877    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"]
Nov 03 19:53:10.383896 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.383851    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"]
Nov 03 19:53:10.387439 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.387416    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:53:10.884873 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.884805    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:10.927755 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.927717    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02097b90-294d-4de0-bb01-c4bb50228769" path="/var/lib/kubelet/pods/02097b90-294d-4de0-bb01-c4bb50228769/volumes"
Nov 03 19:53:10.938318 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.938284    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04530f22-b013-40fe-8390-81a0ea009267" path="/var/lib/kubelet/pods/04530f22-b013-40fe-8390-81a0ea009267/volumes"
Nov 03 19:53:10.944363 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.944333    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23301cd8-4fc8-4544-9744-4fa7cf53ff7c" path="/var/lib/kubelet/pods/23301cd8-4fc8-4544-9744-4fa7cf53ff7c/volumes"
Nov 03 19:53:10.945351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.945326    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a353656-7621-47f3-a298-ef89b4a6ced9" path="/var/lib/kubelet/pods/5a353656-7621-47f3-a298-ef89b4a6ced9/volumes"
Nov 03 19:53:10.946290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.946272    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" path="/var/lib/kubelet/pods/7e1a97c8-d12c-46eb-a635-e450998128d5/volumes"
Nov 03 19:53:10.961058 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:10.961025    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" path="/var/lib/kubelet/pods/bebdbfae-b6b0-4ea1-a180-29de52d1cc60/volumes"
Nov 03 19:53:11.107068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:11.107028    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv" event={"ID":"90e841f9-77b1-4f9e-bf90-a444d25f01d7","Type":"ContainerStarted","Data":"e65d585b0bbb030bc3eaaf5c0be260c5d6bd24e8bf570fa39abd3eb595695c2a"}
Nov 03 19:53:11.109875 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:11.109839    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m" event={"ID":"0d45979f-2173-4f23-ba2d-9ffa2f40e087","Type":"ContainerStarted","Data":"c5dac8959025797f18710a7d2efc6e85c442ca52cbdbbbe899e45eb6710e6db7"}
Nov 03 19:53:11.110160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:11.110123    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m" event={"ID":"0d45979f-2173-4f23-ba2d-9ffa2f40e087","Type":"ContainerStarted","Data":"84727648a2a90e70c637e26bc3cd564f8bec95e31f904999c077986571f12bb7"}
Nov 03 19:53:11.165278 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:11.163206    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m" podStartSLOduration=3.163164407 podStartE2EDuration="3.163164407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:11.129670951 +0000 UTC m=+1304.399794247" watchObservedRunningTime="2026-11-03 19:53:11.163164407 +0000 UTC m=+1304.433287703"
Nov 03 19:53:11.931943 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:11.931904    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:12.131136 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:12.131093    4531 generic.go:356] "Generic (PLEG): container finished" podID="0d45979f-2173-4f23-ba2d-9ffa2f40e087" containerID="c5dac8959025797f18710a7d2efc6e85c442ca52cbdbbbe899e45eb6710e6db7" exitCode=0
Nov 03 19:53:12.131634 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:12.131202    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m" event={"ID":"0d45979f-2173-4f23-ba2d-9ffa2f40e087","Type":"ContainerDied","Data":"c5dac8959025797f18710a7d2efc6e85c442ca52cbdbbbe899e45eb6710e6db7"}
Nov 03 19:53:12.133117 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:12.133093    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv" event={"ID":"90e841f9-77b1-4f9e-bf90-a444d25f01d7","Type":"ContainerStarted","Data":"9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc"}
Nov 03 19:53:12.133265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:12.133151    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:12.182926 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:12.182823    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv" podStartSLOduration=2.706140788 podStartE2EDuration="3.182809932s" totalImagesPullingTime="476.669144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:12.178288298 +0000 UTC m=+1305.448411594" watchObservedRunningTime="2026-11-03 19:53:12.182809932 +0000 UTC m=+1305.452933227"
Nov 03 19:53:12.560343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:12.560258    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"]
Nov 03 19:53:12.560558 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:12.560527    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22" podUID="26a827ee-7cc3-4a48-947f-10b965165761" containerName="maas-api" containerID="cri-o://84e91ca4941c69f08c0ac98d92e44b96b9518b8a32a0083aee8077233a060168" gracePeriod=30
Nov 03 19:53:12.650385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:12.650344    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:12.949199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:12.913636    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:13.198731 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.194139    4531 generic.go:356] "Generic (PLEG): container finished" podID="26a827ee-7cc3-4a48-947f-10b965165761" containerID="84e91ca4941c69f08c0ac98d92e44b96b9518b8a32a0083aee8077233a060168" exitCode=0
Nov 03 19:53:13.198731 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.194570    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22" event={"ID":"26a827ee-7cc3-4a48-947f-10b965165761","Type":"ContainerDied","Data":"84e91ca4941c69f08c0ac98d92e44b96b9518b8a32a0083aee8077233a060168"}
Nov 03 19:53:13.218268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.217765    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:13.219549 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.219405    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:53:13.318198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.318158    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a827ee-7cc3-4a48-947f-10b965165761-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26a827ee-7cc3-4a48-947f-10b965165761-openshift-service-ca\") pod \"26a827ee-7cc3-4a48-947f-10b965165761\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") "
Nov 03 19:53:13.318386 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.318350    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a827ee-7cc3-4a48-947f-10b965165761-kube-api-access-9jmsx\" (UniqueName: \"kubernetes.io/projected/26a827ee-7cc3-4a48-947f-10b965165761-kube-api-access-9jmsx\") pod \"26a827ee-7cc3-4a48-947f-10b965165761\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") "
Nov 03 19:53:13.318455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.318413    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-maas-api-tls\") pod \"26a827ee-7cc3-4a48-947f-10b965165761\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") "
Nov 03 19:53:13.318515 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.318452    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-metrics-certs\") pod \"26a827ee-7cc3-4a48-947f-10b965165761\" (UID: \"26a827ee-7cc3-4a48-947f-10b965165761\") "
Nov 03 19:53:13.318915 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.318888    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a827ee-7cc3-4a48-947f-10b965165761-openshift-service-ca" pod "26a827ee-7cc3-4a48-947f-10b965165761" (UID: "26a827ee-7cc3-4a48-947f-10b965165761"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:13.319591 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.319563    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26a827ee-7cc3-4a48-947f-10b965165761-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.321875 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.321735    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-maas-api-tls" pod "26a827ee-7cc3-4a48-947f-10b965165761" (UID: "26a827ee-7cc3-4a48-947f-10b965165761"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:13.321875 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.321754    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-metrics-certs" pod "26a827ee-7cc3-4a48-947f-10b965165761" (UID: "26a827ee-7cc3-4a48-947f-10b965165761"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:13.321875 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.321799    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a827ee-7cc3-4a48-947f-10b965165761-kube-api-access-9jmsx" pod "26a827ee-7cc3-4a48-947f-10b965165761" (UID: "26a827ee-7cc3-4a48-947f-10b965165761"). InnerVolumeSpecName "kube-api-access-9jmsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:13.404456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.396772    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:13.439258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.439220    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jmsx\" (UniqueName: \"kubernetes.io/projected/26a827ee-7cc3-4a48-947f-10b965165761-kube-api-access-9jmsx\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.439258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.439256    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.439475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.439272    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a827ee-7cc3-4a48-947f-10b965165761-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.453068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.452406    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:13.540192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.540137    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-openshift-ca\") pod \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") "
Nov 03 19:53:13.540406 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.540303    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-xks-ca\") pod \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") "
Nov 03 19:53:13.540519 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.540495    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-openshift-ca" pod "0d45979f-2173-4f23-ba2d-9ffa2f40e087" (UID: "0d45979f-2173-4f23-ba2d-9ffa2f40e087"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:13.540579 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.540537    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d45979f-2173-4f23-ba2d-9ffa2f40e087-kube-api-access-2sfxr\" (UniqueName: \"kubernetes.io/projected/0d45979f-2173-4f23-ba2d-9ffa2f40e087-kube-api-access-2sfxr\") pod \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\" (UID: \"0d45979f-2173-4f23-ba2d-9ffa2f40e087\") "
Nov 03 19:53:13.541349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.541324    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.541613 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.541590    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:13.542691 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.542669    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-xks-ca" pod "0d45979f-2173-4f23-ba2d-9ffa2f40e087" (UID: "0d45979f-2173-4f23-ba2d-9ffa2f40e087"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:13.542963 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.542945    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d45979f-2173-4f23-ba2d-9ffa2f40e087-kube-api-access-2sfxr" pod "0d45979f-2173-4f23-ba2d-9ffa2f40e087" (UID: "0d45979f-2173-4f23-ba2d-9ffa2f40e087"). InnerVolumeSpecName "kube-api-access-2sfxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:13.643001 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.642968    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sfxr\" (UniqueName: \"kubernetes.io/projected/0d45979f-2173-4f23-ba2d-9ffa2f40e087-kube-api-access-2sfxr\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:13.643001 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:13.642996    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d45979f-2173-4f23-ba2d-9ffa2f40e087-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:14.165199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:14.159490    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:14.214010 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:14.213951    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m" event={"ID":"0d45979f-2173-4f23-ba2d-9ffa2f40e087","Type":"ContainerDied","Data":"84727648a2a90e70c637e26bc3cd564f8bec95e31f904999c077986571f12bb7"}
Nov 03 19:53:14.214598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:14.214017    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84727648a2a90e70c637e26bc3cd564f8bec95e31f904999c077986571f12bb7"
Nov 03 19:53:14.214598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:14.214094    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"
Nov 03 19:53:14.219131 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:14.219105    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"
Nov 03 19:53:14.219131 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:14.219119    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22" event={"ID":"26a827ee-7cc3-4a48-947f-10b965165761","Type":"ContainerDied","Data":"4441b7555f3e3f618a0f72a8b64f81676677f5911fb940c2f34ef3fc7d5133b9"}
Nov 03 19:53:14.219322 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:14.219225    4531 scope.go:122] "RemoveContainer" containerID="84e91ca4941c69f08c0ac98d92e44b96b9518b8a32a0083aee8077233a060168"
Nov 03 19:53:14.245965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:14.245923    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"]
Nov 03 19:53:14.250990 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:14.250959    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b73c810-6db878bc98-xqk22"]
Nov 03 19:53:14.773194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:14.772865    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:14.861871 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:14.861837    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a827ee-7cc3-4a48-947f-10b965165761" path="/var/lib/kubelet/pods/26a827ee-7cc3-4a48-947f-10b965165761/volumes"
Nov 03 19:53:15.391862 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:15.391817    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:16.077344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:16.075730    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:16.223864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:16.223824    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:17.046629 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:17.046592    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:17.685199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:17.674980    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:18.201027 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:18.200999    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 19:53:18.469700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:18.469612    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:18.533072 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:18.533036    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 19:53:19.158196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:19.156800    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:19.299634 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:19.299607    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:19.445846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:19.445750    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:19.589088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:19.589060    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:19.936150 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:19.935526    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 19:53:19.947931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:19.945112    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:20.470367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.470327    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"]
Nov 03 19:53:20.471660 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.471631    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy"
Nov 03 19:53:20.471861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.471834    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy"
Nov 03 19:53:20.471973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.471877    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy"
Nov 03 19:53:20.471973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.471885    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy"
Nov 03 19:53:20.471973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.471935    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0d45979f-2173-4f23-ba2d-9ffa2f40e087" containerName="revoke-keys"
Nov 03 19:53:20.471973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.471942    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d45979f-2173-4f23-ba2d-9ffa2f40e087" containerName="revoke-keys"
Nov 03 19:53:20.471973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.471951    4531 cpu_manager.go:414] "containerMap: removing container" podUID="26a827ee-7cc3-4a48-947f-10b965165761" containerName="maas-api"
Nov 03 19:53:20.471973 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.471958    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a827ee-7cc3-4a48-947f-10b965165761" containerName="maas-api"
Nov 03 19:53:20.472328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.472186    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1a97c8-d12c-46eb-a635-e450998128d5" containerName="istio-proxy"
Nov 03 19:53:20.472328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.472201    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bebdbfae-b6b0-4ea1-a180-29de52d1cc60" containerName="istio-proxy"
Nov 03 19:53:20.472328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.472216    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d45979f-2173-4f23-ba2d-9ffa2f40e087" containerName="revoke-keys"
Nov 03 19:53:20.472328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.472264    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a827ee-7cc3-4a48-947f-10b965165761" containerName="maas-api"
Nov 03 19:53:20.504838 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.504760    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"]
Nov 03 19:53:20.505217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.505127    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.510256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.508650    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ffe132-serving-cert\""
Nov 03 19:53:20.578359 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.578323    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:20.609249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.609206    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g24r8\" (UniqueName: \"kubernetes.io/projected/0eb4dd53-79c8-4365-96c7-0a7644c56382-kube-api-access-g24r8\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.612500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.612466    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb4dd53-79c8-4365-96c7-0a7644c56382-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.612788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.612773    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-maas-api-tls\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.613397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.613378    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-metrics-certs\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.679601 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:53:20.679554    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:53:20.721843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.721715    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:20.723782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.723745    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-metrics-certs\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.724542 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.724516    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g24r8\" (UniqueName: \"kubernetes.io/projected/0eb4dd53-79c8-4365-96c7-0a7644c56382-kube-api-access-g24r8\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.734597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.734567    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb4dd53-79c8-4365-96c7-0a7644c56382-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.735915 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.735791    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb4dd53-79c8-4365-96c7-0a7644c56382-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.736160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.735877    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-maas-api-tls\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.743771 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.743740    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-maas-api-tls\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.751245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.746728    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g24r8\" (UniqueName: \"kubernetes.io/projected/0eb4dd53-79c8-4365-96c7-0a7644c56382-kube-api-access-g24r8\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.756771 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.756718    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-metrics-certs\") pod \"maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.840267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.839952    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:20.909441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:20.909410    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:21.069791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:21.069763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:21.419264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:21.416536    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"]
Nov 03 19:53:21.422366 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:21.422327    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb4dd53_79c8_4365_96c7_0a7644c56382.slice/crio-304e6f43b883501952e9889843cd82eef168e04699cb9f0a644cb6b34b04077d WatchSource:0}: Error finding container 304e6f43b883501952e9889843cd82eef168e04699cb9f0a644cb6b34b04077d: Status 404 returned error can't find the container with id 304e6f43b883501952e9889843cd82eef168e04699cb9f0a644cb6b34b04077d
Nov 03 19:53:21.829475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:21.829384    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:21.987608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:21.985836    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:22.176513 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.176020    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:22.199710 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.199256    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"]
Nov 03 19:53:22.205813 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.205781    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.244456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.244424    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"]
Nov 03 19:53:22.244643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.244624    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0c8eec-openshift-default-dockercfg-ns84n\""
Nov 03 19:53:22.460196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.459282    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-podinfo\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.460196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.459360    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-socket\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.460196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.459473    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-data\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.460196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.459558    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-credential-socket\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.460196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.459659    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-token\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.460196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.459775    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-certs\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.460196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.459837    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istiod-ca-cert\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.460196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.459889    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pdjm\" (UniqueName: \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-kube-api-access-5pdjm\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.460196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.460104    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-envoy\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.656332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.656279    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-certs\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.657853 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.657628    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:22.659471 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.658041    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istiod-ca-cert\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.659799 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.659220    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istiod-ca-cert\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.672366 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.672303    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-certs\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.674576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.674544    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdjm\" (UniqueName: \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-kube-api-access-5pdjm\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.675049 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.675012    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-envoy\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.675275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.675255    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-podinfo\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.675427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.675414    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-socket\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.675608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.675592    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-data\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.675752 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.675739    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-credential-socket\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.675902 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.675890    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-token\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.684052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.683906    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-socket\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.699601 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.688240    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-data\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.699870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.688562    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-credential-socket\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.785561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.785480    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-token\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.787085 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.786956    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pdjm\" (UniqueName: \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-kube-api-access-5pdjm\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.789938 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.789912    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-podinfo\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.792953 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.792768    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh" event={"ID":"0eb4dd53-79c8-4365-96c7-0a7644c56382","Type":"ContainerStarted","Data":"6473d02b53b2dd8a8ef3e926e6cb0110df4d44612b81fec6f4d3afcb6ee2b737"}
Nov 03 19:53:22.792953 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.792864    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh" event={"ID":"0eb4dd53-79c8-4365-96c7-0a7644c56382","Type":"ContainerStarted","Data":"304e6f43b883501952e9889843cd82eef168e04699cb9f0a644cb6b34b04077d"}
Nov 03 19:53:22.792953 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.792917    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:22.796419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.795771    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-envoy\") pod \"e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") " pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:22.799168 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.799148    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:22.814396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.814340    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh" podStartSLOduration=2.395366145 podStartE2EDuration="2.814323524s" totalImagesPullingTime="418.957379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:22.809034122 +0000 UTC m=+1316.079157416" watchObservedRunningTime="2026-11-03 19:53:22.814323524 +0000 UTC m=+1316.084446821"
Nov 03 19:53:22.864896 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:22.864871    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:23.180999 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:23.180960    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:23.367870 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:23.367838    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9ec0ed_4670_47ca_aa89_dd22c8bd9754.slice/crio-1ea17365ee7c30b1dded3afd2b501ee11111c956e53c9d58a12c6ff25ee0b67b WatchSource:0}: Error finding container 1ea17365ee7c30b1dded3afd2b501ee11111c956e53c9d58a12c6ff25ee0b67b: Status 404 returned error can't find the container with id 1ea17365ee7c30b1dded3afd2b501ee11111c956e53c9d58a12c6ff25ee0b67b
Nov 03 19:53:23.368634 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:23.368605    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"]
Nov 03 19:53:23.371032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:23.370995    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:23.371212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:23.371077    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:23.371212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:23.371125    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:23.772098 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:23.771069    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:23.889645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:23.889610    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" event={"ID":"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754","Type":"ContainerStarted","Data":"282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5"}
Nov 03 19:53:23.890258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:23.889669    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" event={"ID":"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754","Type":"ContainerStarted","Data":"1ea17365ee7c30b1dded3afd2b501ee11111c956e53c9d58a12c6ff25ee0b67b"}
Nov 03 19:53:23.907951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:23.907887    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" podStartSLOduration=1.907869169 podStartE2EDuration="1.907869169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:23.906300151 +0000 UTC m=+1317.176423446" watchObservedRunningTime="2026-11-03 19:53:23.907869169 +0000 UTC m=+1317.177992464"
Nov 03 19:53:24.815821 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:24.815783    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:24.865164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:24.865134    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:24.871339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:24.871314    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:24.910428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:24.910384    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:24.911642 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:24.911619    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:53:25.770198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:25.763731    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:26.253931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:26.253899    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:26.392054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:26.392025    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:27.088199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.078388    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:27.462008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.458873    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"]
Nov 03 19:53:27.466833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.466803    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:53:27.467005 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.466881    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:27.469750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.469724    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"]
Nov 03 19:53:27.474927 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.474884    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0c8eec-dockercfg-q56ng\""
Nov 03 19:53:27.475218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.475146    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0c8eec\""
Nov 03 19:53:27.494312 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.494284    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"]
Nov 03 19:53:27.500153 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.500127    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:53:27.510097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.510066    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"]
Nov 03 19:53:27.561843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.561695    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5555cb-2681-4a70-af23-ba1281bddc74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8\" (UID: \"7a5555cb-2681-4a70-af23-ba1281bddc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:53:27.561843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.561824    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a527b3f0-9f65-4d15-b849-432b1bb3e445-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt\" (UID: \"a527b3f0-9f65-4d15-b849-432b1bb3e445\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:53:27.562534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.562146    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mstg\" (UniqueName: \"kubernetes.io/projected/7a5555cb-2681-4a70-af23-ba1281bddc74-kube-api-access-5mstg\") pod \"payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8\" (UID: \"7a5555cb-2681-4a70-af23-ba1281bddc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:53:27.562534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.562212    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4sq\" (UniqueName: \"kubernetes.io/projected/a527b3f0-9f65-4d15-b849-432b1bb3e445-kube-api-access-7c4sq\") pod \"payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt\" (UID: \"a527b3f0-9f65-4d15-b849-432b1bb3e445\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:53:27.639309 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.621532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:27.685504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.685466    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5555cb-2681-4a70-af23-ba1281bddc74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8\" (UID: \"7a5555cb-2681-4a70-af23-ba1281bddc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:53:27.686773 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.686745    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a527b3f0-9f65-4d15-b849-432b1bb3e445-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt\" (UID: \"a527b3f0-9f65-4d15-b849-432b1bb3e445\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:53:27.692305 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.686607    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5555cb-2681-4a70-af23-ba1281bddc74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8\" (UID: \"7a5555cb-2681-4a70-af23-ba1281bddc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:53:27.692470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.687701    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a527b3f0-9f65-4d15-b849-432b1bb3e445-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt\" (UID: \"a527b3f0-9f65-4d15-b849-432b1bb3e445\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:53:27.692661 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.692642    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mstg\" (UniqueName: \"kubernetes.io/projected/7a5555cb-2681-4a70-af23-ba1281bddc74-kube-api-access-5mstg\") pod \"payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8\" (UID: \"7a5555cb-2681-4a70-af23-ba1281bddc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:53:27.696623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.696599    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4sq\" (UniqueName: \"kubernetes.io/projected/a527b3f0-9f65-4d15-b849-432b1bb3e445-kube-api-access-7c4sq\") pod \"payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt\" (UID: \"a527b3f0-9f65-4d15-b849-432b1bb3e445\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:53:27.720090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.720011    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4sq\" (UniqueName: \"kubernetes.io/projected/a527b3f0-9f65-4d15-b849-432b1bb3e445-kube-api-access-7c4sq\") pod \"payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt\" (UID: \"a527b3f0-9f65-4d15-b849-432b1bb3e445\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:53:27.734063 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.733983    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mstg\" (UniqueName: \"kubernetes.io/projected/7a5555cb-2681-4a70-af23-ba1281bddc74-kube-api-access-5mstg\") pod \"payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8\" (UID: \"7a5555cb-2681-4a70-af23-ba1281bddc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:53:27.838216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.824222    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:53:27.838216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:27.825200    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:53:28.281841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:28.281804    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"]
Nov 03 19:53:28.281935 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:28.281881    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5555cb_2681_4a70_af23_ba1281bddc74.slice/crio-db86b72c75096f541afe9d80d8d55ff9cd77ce04e9220fab4f93cf435119fe86 WatchSource:0}: Error finding container db86b72c75096f541afe9d80d8d55ff9cd77ce04e9220fab4f93cf435119fe86: Status 404 returned error can't find the container with id db86b72c75096f541afe9d80d8d55ff9cd77ce04e9220fab4f93cf435119fe86
Nov 03 19:53:28.284269 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:28.284243    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda527b3f0_9f65_4d15_b849_432b1bb3e445.slice/crio-c2319c7968e7cf7e04d4fc42f6db8f9a70309c70b8884162eafa7b814d347edf WatchSource:0}: Error finding container c2319c7968e7cf7e04d4fc42f6db8f9a70309c70b8884162eafa7b814d347edf: Status 404 returned error can't find the container with id c2319c7968e7cf7e04d4fc42f6db8f9a70309c70b8884162eafa7b814d347edf
Nov 03 19:53:28.286819 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:28.286797    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"]
Nov 03 19:53:28.412193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:28.411218    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:28.542471 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:28.542390    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:28.676104 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:28.676069    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:28.971198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:28.967481    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 19:53:29.048769 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:29.048733    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8" event={"ID":"7a5555cb-2681-4a70-af23-ba1281bddc74","Type":"ContainerStarted","Data":"3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37"}
Nov 03 19:53:29.049647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:29.049532    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8" event={"ID":"7a5555cb-2681-4a70-af23-ba1281bddc74","Type":"ContainerStarted","Data":"db86b72c75096f541afe9d80d8d55ff9cd77ce04e9220fab4f93cf435119fe86"}
Nov 03 19:53:29.050077 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:29.049997    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:53:29.052538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:29.052485    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt" event={"ID":"a527b3f0-9f65-4d15-b849-432b1bb3e445","Type":"ContainerStarted","Data":"6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251"}
Nov 03 19:53:29.052538 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:29.052535    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt" event={"ID":"a527b3f0-9f65-4d15-b849-432b1bb3e445","Type":"ContainerStarted","Data":"c2319c7968e7cf7e04d4fc42f6db8f9a70309c70b8884162eafa7b814d347edf"}
Nov 03 19:53:29.053532 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:29.052668    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:53:29.067353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:29.067305    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8" podStartSLOduration=2.067293248 podStartE2EDuration="2.067293248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:29.06707126 +0000 UTC m=+1322.337194562" watchObservedRunningTime="2026-11-03 19:53:29.067293248 +0000 UTC m=+1322.337416543"
Nov 03 19:53:29.090664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:29.090604    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt" podStartSLOduration=2.090587653 podStartE2EDuration="2.090587653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:29.085786859 +0000 UTC m=+1322.355910157" watchObservedRunningTime="2026-11-03 19:53:29.090587653 +0000 UTC m=+1322.360710950"
Nov 03 19:53:29.938157 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:29.938111    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:30.258290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:30.258160    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:30.582489 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:30.582389    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:30.749765 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:30.749726    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:31.396196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:31.380523    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:31.792867 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:31.792766    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:32.158838 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:32.158776    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:32.291952 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:32.291904    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:32.644614 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:32.644585    4531 scope.go:122] "RemoveContainer" containerID="733f275c337c04f85fe43381d59bd49d976e1815f6856bdc41da9333acbfd9d4"
Nov 03 19:53:32.675144 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:32.675117    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:32.675512 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:32.675493    4531 scope.go:122] "RemoveContainer" containerID="2006cc46390ab1b2e044c64e36cfd398534a4688495cd14f55e3181c7aec2177"
Nov 03 19:53:32.704542 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:32.692919    4531 scope.go:122] "RemoveContainer" containerID="7bdc6c802ee84a02a0b5039e98498f9aeb82d6e99aafdbdfa1e7b7cc9dab2508"
Nov 03 19:53:32.903482 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:32.902726    4531 scope.go:122] "RemoveContainer" containerID="5a612058689b69f6b24376441ef0ea9c4091a08f7ee202c471b93e95b55e1010"
Nov 03 19:53:33.004015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:32.969812    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:33.139280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.137987    4531 scope.go:122] "RemoveContainer" containerID="6d7b41d9d0523457e2d7fb576944aca810853eb8385b3c798a6ebbbf0149cc68"
Nov 03 19:53:33.144638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.144602    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:33.160279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.160258    4531 scope.go:122] "RemoveContainer" containerID="e76efef59cfc1f5c9efaf53b67992e5c96111de47729d4053c536ac5bd7049c2"
Nov 03 19:53:33.174803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.174766    4531 scope.go:122] "RemoveContainer" containerID="1cf700c12ef41f0b5d53b293b021cadf954c45e735081b8f3c81b7533cac6a6d"
Nov 03 19:53:33.186562 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.186543    4531 scope.go:122] "RemoveContainer" containerID="e8fc30bc626ec738309997f18dbc2ff1c9523f34ae648227a481ef2da6fd2ac8"
Nov 03 19:53:33.199729 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.199665    4531 scope.go:122] "RemoveContainer" containerID="f8786709590d5108fb93cb8e810f40c82bf288ef063a5b747e4a92a8c33512f8"
Nov 03 19:53:33.216293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.216272    4531 scope.go:122] "RemoveContainer" containerID="c49fbba7f272ced5bceca820aee11cb53eff1272744a68d5c9679c570b8b7f38"
Nov 03 19:53:33.227268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.227244    4531 scope.go:122] "RemoveContainer" containerID="79b0f17a6e486ca59be232dfa73ee4f8d950772f18d7f710e6d6c10b8ea0b35d"
Nov 03 19:53:33.238540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.238517    4531 scope.go:122] "RemoveContainer" containerID="b6c5759d4f1e36d4e356d0cb32a88bac712df7f4dc6d78a04078896b8cece468"
Nov 03 19:53:33.336034 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.334419    4531 scope.go:122] "RemoveContainer" containerID="aa903fa27798fb32d84009cb02b3956f87643123c504f48e019272d32d02921f"
Nov 03 19:53:33.337954 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.337897    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:33.362221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.361262    4531 scope.go:122] "RemoveContainer" containerID="2a0abb67cc9fabaf06a54962c6af49dfb91f64d3ded0fdb4c21de95dd4ea2659"
Nov 03 19:53:33.383489 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.383466    4531 scope.go:122] "RemoveContainer" containerID="f53f7a497a8f91b32b9f630ec40eb6995997df6641adbe53660ad828e6f296b1"
Nov 03 19:53:33.394005 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.393986    4531 scope.go:122] "RemoveContainer" containerID="dd025fbe7dd72be54b2e26c747cb23b7f40ad07a3f36594fec879ed5ae878978"
Nov 03 19:53:33.621855 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.621813    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:33.787987 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:33.787898    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:34.152200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:34.144213    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:34.651242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:34.640205    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:35.247304 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:35.247053    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:36.436229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:36.429684    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:36.835706 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:36.835633    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:36.971160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:36.971122    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:37.608866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:37.608770    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:37.771102 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:37.771068    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:37.925245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:37.925211    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:38.166788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.166750    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:38.188506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.188434    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"]
Nov 03 19:53:38.200652 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.200622    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 19:53:38.201504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.201479    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"]
Nov 03 19:53:38.212540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.212512    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-332694\""
Nov 03 19:53:38.212695 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.212558    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-332694-dockercfg-cnbcv\""
Nov 03 19:53:38.258735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.258700    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"]
Nov 03 19:53:38.264306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.264276    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 19:53:38.331213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.309357    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8sjz\" (UniqueName: \"kubernetes.io/projected/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-kube-api-access-b8sjz\") pod \"payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr\" (UID: \"db439ef2-1c8e-4969-8c07-78c7ec87c3d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 19:53:38.331768 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.331729    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd\" (UID: \"6551edf5-a4a6-4b4d-bb17-1b32b335c58e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 19:53:38.332054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.332034    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr\" (UID: \"db439ef2-1c8e-4969-8c07-78c7ec87c3d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 19:53:38.332197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.332164    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmt5\" (UniqueName: \"kubernetes.io/projected/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-kube-api-access-2zmt5\") pod \"payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd\" (UID: \"6551edf5-a4a6-4b4d-bb17-1b32b335c58e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 19:53:38.373195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.366229    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"]
Nov 03 19:53:38.386434 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.386306    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:38.547775 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.547684    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8sjz\" (UniqueName: \"kubernetes.io/projected/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-kube-api-access-b8sjz\") pod \"payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr\" (UID: \"db439ef2-1c8e-4969-8c07-78c7ec87c3d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 19:53:38.549017 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.548932    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd\" (UID: \"6551edf5-a4a6-4b4d-bb17-1b32b335c58e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 19:53:38.550441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.550416    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr\" (UID: \"db439ef2-1c8e-4969-8c07-78c7ec87c3d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 19:53:38.552117 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.552096    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmt5\" (UniqueName: \"kubernetes.io/projected/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-kube-api-access-2zmt5\") pod \"payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd\" (UID: \"6551edf5-a4a6-4b4d-bb17-1b32b335c58e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 19:53:38.552778 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.550115    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd\" (UID: \"6551edf5-a4a6-4b4d-bb17-1b32b335c58e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 19:53:38.553704 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.551432    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr\" (UID: \"db439ef2-1c8e-4969-8c07-78c7ec87c3d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 19:53:38.564048 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.563986    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8sjz\" (UniqueName: \"kubernetes.io/projected/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-kube-api-access-b8sjz\") pod \"payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr\" (UID: \"db439ef2-1c8e-4969-8c07-78c7ec87c3d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 19:53:38.608202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.589659    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmt5\" (UniqueName: \"kubernetes.io/projected/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-kube-api-access-2zmt5\") pod \"payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd\" (UID: \"6551edf5-a4a6-4b4d-bb17-1b32b335c58e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 19:53:38.824119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.824030    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 19:53:38.860822 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:38.860782    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 19:53:39.127628 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:39.127590    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6551edf5_a4a6_4b4d_bb17_1b32b335c58e.slice/crio-6b69122991b5ff18d9f8d4f8c1f86ed4c8a3df5f5c8e672f573f6b5e58998be2 WatchSource:0}: Error finding container 6b69122991b5ff18d9f8d4f8c1f86ed4c8a3df5f5c8e672f573f6b5e58998be2: Status 404 returned error can't find the container with id 6b69122991b5ff18d9f8d4f8c1f86ed4c8a3df5f5c8e672f573f6b5e58998be2
Nov 03 19:53:39.127733 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.127658    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"]
Nov 03 19:53:39.200427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.197906    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:39.268919 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.268870    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"]
Nov 03 19:53:39.274167 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:39.274139    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb439ef2_1c8e_4969_8c07_78c7ec87c3d9.slice/crio-94e5f64318d4521826373717a2a7c964d0b95e805898b92f6c8f90826e14fb19 WatchSource:0}: Error finding container 94e5f64318d4521826373717a2a7c964d0b95e805898b92f6c8f90826e14fb19: Status 404 returned error can't find the container with id 94e5f64318d4521826373717a2a7c964d0b95e805898b92f6c8f90826e14fb19
Nov 03 19:53:39.274655 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.274623    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd" event={"ID":"6551edf5-a4a6-4b4d-bb17-1b32b335c58e","Type":"ContainerStarted","Data":"6b69122991b5ff18d9f8d4f8c1f86ed4c8a3df5f5c8e672f573f6b5e58998be2"}
Nov 03 19:53:39.533019 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.532982    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:39.535555 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.535533    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"]
Nov 03 19:53:39.541014 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.540993    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 19:53:39.546403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.545562    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-32a584\""
Nov 03 19:53:39.560529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.560509    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"]
Nov 03 19:53:39.583413 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.583383    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"]
Nov 03 19:53:39.588613 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.588587    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 19:53:39.594439 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.594405    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"]
Nov 03 19:53:39.684193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.683938    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-32a584-dockercfg-f5m78\""
Nov 03 19:53:39.698382 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.690007    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:39.714207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.704600    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccw9t\" (UniqueName: \"kubernetes.io/projected/6c5f6c12-d596-48e5-90bc-cc0f523a106b-kube-api-access-ccw9t\") pod \"payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk\" (UID: \"6c5f6c12-d596-48e5-90bc-cc0f523a106b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 19:53:39.714207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.704696    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb\" (UID: \"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 19:53:39.714207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.705011    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlvh\" (UniqueName: \"kubernetes.io/projected/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-kube-api-access-tdlvh\") pod \"payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb\" (UID: \"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 19:53:39.714207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.705084    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk\" (UID: \"6c5f6c12-d596-48e5-90bc-cc0f523a106b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 19:53:39.816209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.813165    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlvh\" (UniqueName: \"kubernetes.io/projected/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-kube-api-access-tdlvh\") pod \"payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb\" (UID: \"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 19:53:39.816209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.813265    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk\" (UID: \"6c5f6c12-d596-48e5-90bc-cc0f523a106b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 19:53:39.816209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.813425    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccw9t\" (UniqueName: \"kubernetes.io/projected/6c5f6c12-d596-48e5-90bc-cc0f523a106b-kube-api-access-ccw9t\") pod \"payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk\" (UID: \"6c5f6c12-d596-48e5-90bc-cc0f523a106b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 19:53:39.816209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.813493    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb\" (UID: \"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 19:53:39.816209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.814608    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb\" (UID: \"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 19:53:39.816209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.815717    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk\" (UID: \"6c5f6c12-d596-48e5-90bc-cc0f523a106b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 19:53:39.868209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.865806    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlvh\" (UniqueName: \"kubernetes.io/projected/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-kube-api-access-tdlvh\") pod \"payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb\" (UID: \"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 19:53:39.926022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.925437    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 19:53:39.934141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:39.934103    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccw9t\" (UniqueName: \"kubernetes.io/projected/6c5f6c12-d596-48e5-90bc-cc0f523a106b-kube-api-access-ccw9t\") pod \"payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk\" (UID: \"6c5f6c12-d596-48e5-90bc-cc0f523a106b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 19:53:40.065149 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.065122    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:53:40.164000 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.163842    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 19:53:40.286346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.286313    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr" event={"ID":"db439ef2-1c8e-4969-8c07-78c7ec87c3d9","Type":"ContainerStarted","Data":"11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6"}
Nov 03 19:53:40.286554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.286376    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr" event={"ID":"db439ef2-1c8e-4969-8c07-78c7ec87c3d9","Type":"ContainerStarted","Data":"94e5f64318d4521826373717a2a7c964d0b95e805898b92f6c8f90826e14fb19"}
Nov 03 19:53:40.286554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.286422    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 19:53:40.288695 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.288665    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd" event={"ID":"6551edf5-a4a6-4b4d-bb17-1b32b335c58e","Type":"ContainerStarted","Data":"003266267a5342a6d8a2c2ff281030e9b303fb99e2bbe031ce333a9f90e313dd"}
Nov 03 19:53:40.288852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.288732    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 19:53:40.301912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.301858    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr" podStartSLOduration=2.301842557 podStartE2EDuration="2.301842557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:40.300211883 +0000 UTC m=+1333.570335174" watchObservedRunningTime="2026-11-03 19:53:40.301842557 +0000 UTC m=+1333.571965851"
Nov 03 19:53:40.312360 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:40.312321    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6a64a2_11cd_4280_b96a_ee0985f8ebe8.slice/crio-ced4b1fb83a70420d12c4ea72746b25764d2b31edd00044f14cd2d7f86755fcc WatchSource:0}: Error finding container ced4b1fb83a70420d12c4ea72746b25764d2b31edd00044f14cd2d7f86755fcc: Status 404 returned error can't find the container with id ced4b1fb83a70420d12c4ea72746b25764d2b31edd00044f14cd2d7f86755fcc
Nov 03 19:53:40.317025 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.316997    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"]
Nov 03 19:53:40.319455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.319403    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd" podStartSLOduration=2.3193879219999998 podStartE2EDuration="2.319387922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:40.315457217 +0000 UTC m=+1333.585580511" watchObservedRunningTime="2026-11-03 19:53:40.319387922 +0000 UTC m=+1333.589511217"
Nov 03 19:53:40.505225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.505133    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:40.519409 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:40.506254    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5f6c12_d596_48e5_90bc_cc0f523a106b.slice/crio-366b43bfeb7c7d9783c322effc21d3c8018f9a347ac60a80c3b91daf76e13860 WatchSource:0}: Error finding container 366b43bfeb7c7d9783c322effc21d3c8018f9a347ac60a80c3b91daf76e13860: Status 404 returned error can't find the container with id 366b43bfeb7c7d9783c322effc21d3c8018f9a347ac60a80c3b91daf76e13860
Nov 03 19:53:40.543198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:40.532483    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"]
Nov 03 19:53:41.108435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:41.108398    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:53:41.399164 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:41.399125    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb" event={"ID":"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8","Type":"ContainerStarted","Data":"336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3"}
Nov 03 19:53:41.399411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:41.399205    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb" event={"ID":"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8","Type":"ContainerStarted","Data":"ced4b1fb83a70420d12c4ea72746b25764d2b31edd00044f14cd2d7f86755fcc"}
Nov 03 19:53:41.399411 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:41.399318    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 19:53:41.401032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:41.400999    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk" event={"ID":"6c5f6c12-d596-48e5-90bc-cc0f523a106b","Type":"ContainerStarted","Data":"39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a"}
Nov 03 19:53:41.401163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:41.401060    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk" event={"ID":"6c5f6c12-d596-48e5-90bc-cc0f523a106b","Type":"ContainerStarted","Data":"366b43bfeb7c7d9783c322effc21d3c8018f9a347ac60a80c3b91daf76e13860"}
Nov 03 19:53:41.401163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:41.401104    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 19:53:41.425054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:41.424988    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb" podStartSLOduration=2.424967238 podStartE2EDuration="2.424967238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:41.41347855 +0000 UTC m=+1334.683601850" watchObservedRunningTime="2026-11-03 19:53:41.424967238 +0000 UTC m=+1334.695090538"
Nov 03 19:53:41.440832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:41.440766    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk" podStartSLOduration=2.440745362 podStartE2EDuration="2.440745362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:41.431921887 +0000 UTC m=+1334.702045183" watchObservedRunningTime="2026-11-03 19:53:41.440745362 +0000 UTC m=+1334.710868658"
Nov 03 19:53:41.531337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:41.531301    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:42.417531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:42.417493    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:43.250197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:43.247826    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:43.869195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:43.860258    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:44.764014 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:44.763981    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:44.980119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:44.980086    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:45.141667 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:45.141635    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:45.491705 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:45.491616    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:46.020097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:46.020048    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:46.165194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:46.163080    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:46.574818 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:46.574768    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:47.062117 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:47.062072    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:47.493802 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:47.493762    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:47.885249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:47.880819    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:49.031307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:49.027471    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:49.463197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:49.461079    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:49.597457 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:49.597425    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:50.179196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.179139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:50.591117 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:53:50.591030    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:53:50.801084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.801047    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"]
Nov 03 19:53:50.806284 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.806258    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 19:53:50.810494 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.810451    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a99fad\""
Nov 03 19:53:50.813441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.813411    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:50.814879 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.814858    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"]
Nov 03 19:53:50.880167 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.880117    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"]
Nov 03 19:53:50.885128 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.885102    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"]
Nov 03 19:53:50.885315 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.885217    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 19:53:50.888355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.888334    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a99fad-dockercfg-vswlc\""
Nov 03 19:53:50.958196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.947913    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82pb\" (UniqueName: \"kubernetes.io/projected/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-kube-api-access-h82pb\") pod \"payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj\" (UID: \"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 19:53:50.958196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.948049    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj\" (UID: \"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 19:53:50.995989 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:50.995948    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:51.091229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.057002    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35af209c-d9d9-4821-a124-2183e18a84c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6\" (UID: \"35af209c-d9d9-4821-a124-2183e18a84c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 19:53:51.091229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.057087    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrh88\" (UniqueName: \"kubernetes.io/projected/35af209c-d9d9-4821-a124-2183e18a84c5-kube-api-access-zrh88\") pod \"payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6\" (UID: \"35af209c-d9d9-4821-a124-2183e18a84c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 19:53:51.091229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.057121    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82pb\" (UniqueName: \"kubernetes.io/projected/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-kube-api-access-h82pb\") pod \"payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj\" (UID: \"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 19:53:51.091229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.057272    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj\" (UID: \"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 19:53:51.092483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.092452    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj\" (UID: \"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 19:53:51.120420 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.120355    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82pb\" (UniqueName: \"kubernetes.io/projected/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-kube-api-access-h82pb\") pod \"payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj\" (UID: \"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 19:53:51.132818 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.132492    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 19:53:51.209750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.193054    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35af209c-d9d9-4821-a124-2183e18a84c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6\" (UID: \"35af209c-d9d9-4821-a124-2183e18a84c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 19:53:51.209750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.193146    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrh88\" (UniqueName: \"kubernetes.io/projected/35af209c-d9d9-4821-a124-2183e18a84c5-kube-api-access-zrh88\") pod \"payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6\" (UID: \"35af209c-d9d9-4821-a124-2183e18a84c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 19:53:51.212291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.212258    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35af209c-d9d9-4821-a124-2183e18a84c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6\" (UID: \"35af209c-d9d9-4821-a124-2183e18a84c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 19:53:51.263069 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.262995    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrh88\" (UniqueName: \"kubernetes.io/projected/35af209c-d9d9-4821-a124-2183e18a84c5-kube-api-access-zrh88\") pod \"payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6\" (UID: \"35af209c-d9d9-4821-a124-2183e18a84c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 19:53:51.404461 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.404379    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 19:53:51.405945 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.405919    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 19:53:51.504193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.504148    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 19:53:51.634441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.634404    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:51.738351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.734861    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"]
Nov 03 19:53:51.827234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:51.820666    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:52.029201 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:53:52.029148    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35af209c_d9d9_4821_a124_2183e18a84c5.slice/crio-b953f9603618e762073aa97623262a9a419213fd01529ed93821883b2957a809 WatchSource:0}: Error finding container b953f9603618e762073aa97623262a9a419213fd01529ed93821883b2957a809: Status 404 returned error can't find the container with id b953f9603618e762073aa97623262a9a419213fd01529ed93821883b2957a809
Nov 03 19:53:52.051737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.051698    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"]
Nov 03 19:53:52.470216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.458594    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 19:53:52.479208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.473446    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 19:53:52.498045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.492776    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:52.676497 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.676452    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj" event={"ID":"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92","Type":"ContainerStarted","Data":"0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa"}
Nov 03 19:53:52.676664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.676529    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj" event={"ID":"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92","Type":"ContainerStarted","Data":"57429b1d37fd954553d938dbac74fb40296350ed38e7e38c2787b48c9b6052ed"}
Nov 03 19:53:52.676664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.676560    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 19:53:52.678344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.678315    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6" event={"ID":"35af209c-d9d9-4821-a124-2183e18a84c5","Type":"ContainerStarted","Data":"8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0"}
Nov 03 19:53:52.678476 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.678366    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 19:53:52.678476 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.678382    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6" event={"ID":"35af209c-d9d9-4821-a124-2183e18a84c5","Type":"ContainerStarted","Data":"b953f9603618e762073aa97623262a9a419213fd01529ed93821883b2957a809"}
Nov 03 19:53:52.694738 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.694687    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj" podStartSLOduration=2.694675461 podStartE2EDuration="2.694675461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:52.691799398 +0000 UTC m=+1345.961922694" watchObservedRunningTime="2026-11-03 19:53:52.694675461 +0000 UTC m=+1345.964798767"
Nov 03 19:53:52.711286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.711235    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6" podStartSLOduration=2.711221928 podStartE2EDuration="2.711221928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:52.706634395 +0000 UTC m=+1345.976757691" watchObservedRunningTime="2026-11-03 19:53:52.711221928 +0000 UTC m=+1345.981345252"
Nov 03 19:53:52.836645 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:52.836510    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:53.260198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:53.259784    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:53.936568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:53.936536    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:54.763691 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:54.755409    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:55.430254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:55.430218    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:55.584129 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:55.583999    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:56.235864 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:56.235824    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:56.468016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:56.467971    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:56.489543 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:56.489414    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"]
Nov 03 19:53:56.564419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:56.562932    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f84de-xnt5m"]
Nov 03 19:53:56.659826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:56.659736    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:56.964567 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:56.964530    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d45979f-2173-4f23-ba2d-9ffa2f40e087" path="/var/lib/kubelet/pods/0d45979f-2173-4f23-ba2d-9ffa2f40e087/volumes"
Nov 03 19:53:57.029110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:57.029029    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:57.756138 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:57.756111    4531 generic.go:356] "Generic (PLEG): container finished" podID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerID="3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee" exitCode=0
Nov 03 19:53:57.756651 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:57.756198    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" event={"ID":"505a4164-fe42-4574-aeb6-fbe39f93e798","Type":"ContainerDied","Data":"3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee"}
Nov 03 19:53:57.756783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:57.756767    4531 scope.go:122] "RemoveContainer" containerID="3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee"
Nov 03 19:53:57.759549 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:57.759495    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:57.759727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:57.759591    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:57.759727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:57.759646    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:58.150762 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:58.150728    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:58.317863 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:58.317818    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:58.497035 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:58.496956    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:58.835261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:58.831999    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" event={"ID":"505a4164-fe42-4574-aeb6-fbe39f93e798","Type":"ContainerStarted","Data":"be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052"}
Nov 03 19:53:59.411196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:59.411092    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:59.671254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:59.663740    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:53:59.686238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:59.677213    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 19:53:59.686238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:53:59.677294    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 19:54:00.482205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:00.476419    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:00.658198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:00.656578    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 19:54:00.658198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:00.656665    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 19:54:01.369666 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:01.369624    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:01.662344 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:01.662301    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 19:54:01.662788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:01.662366    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 19:54:02.135930 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:02.130281    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:02.659201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:02.657593    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 19:54:02.659201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:02.657670    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 19:54:03.022568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:03.020146    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:03.224739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:03.224707    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:54:03.330541 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:03.330454    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:03.474431 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:03.474391    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:03.661588 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:03.661547    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 19:54:03.661812 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:03.661608    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 19:54:03.683350 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:03.683313    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:03.716429 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:03.716393    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 19:54:03.754403 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:03.754373    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 19:54:04.293539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.293491    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:04.354145 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.354100    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"]
Nov 03 19:54:04.390637 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.390604    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"]
Nov 03 19:54:04.390806 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.390709    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:04.393385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.393366    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:54:04.393647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.393628    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:54:04.506094 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.506062    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:04.506312 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.506107    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn2sw\" (UniqueName: \"kubernetes.io/projected/636635fb-6bb9-4b30-bf3f-e3650a20a231-kube-api-access-gn2sw\") pod \"maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:04.506312 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.506144    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:04.609015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.608923    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:04.609015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.608982    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2sw\" (UniqueName: \"kubernetes.io/projected/636635fb-6bb9-4b30-bf3f-e3650a20a231-kube-api-access-gn2sw\") pod \"maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:04.609250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.609031    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:04.609765 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.609740    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:04.611479 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.611452    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:04.617865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.617841    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2sw\" (UniqueName: \"kubernetes.io/projected/636635fb-6bb9-4b30-bf3f-e3650a20a231-kube-api-access-gn2sw\") pod \"maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:04.664351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.664311    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 19:54:04.664608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.664378    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 19:54:04.707782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:04.707752    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:05.018466 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:54:05.018418    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636635fb_6bb9_4b30_bf3f_e3650a20a231.slice/crio-95400191f3e6be811ddcbe0154868d66876c366d142b6344a5c5e72326a44ff0 WatchSource:0}: Error finding container 95400191f3e6be811ddcbe0154868d66876c366d142b6344a5c5e72326a44ff0: Status 404 returned error can't find the container with id 95400191f3e6be811ddcbe0154868d66876c366d142b6344a5c5e72326a44ff0
Nov 03 19:54:05.019753 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:05.019723    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"]
Nov 03 19:54:05.319082 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:05.318992    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:05.407195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:05.406582    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc" event={"ID":"636635fb-6bb9-4b30-bf3f-e3650a20a231","Type":"ContainerStarted","Data":"95400191f3e6be811ddcbe0154868d66876c366d142b6344a5c5e72326a44ff0"}
Nov 03 19:54:05.660549 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:05.660521    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:54:05.661707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:05.661682    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 19:54:06.410270 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:06.410240    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:06.492204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:06.472102    4531 generic.go:356] "Generic (PLEG): container finished" podID="636635fb-6bb9-4b30-bf3f-e3650a20a231" containerID="0a6c31d1d5bb15e41d4c15e78362f07e8aa722fda94e0cc51db8e1a82aed8a4d" exitCode=0
Nov 03 19:54:06.493236 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:06.493199    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc" event={"ID":"636635fb-6bb9-4b30-bf3f-e3650a20a231","Type":"ContainerDied","Data":"0a6c31d1d5bb15e41d4c15e78362f07e8aa722fda94e0cc51db8e1a82aed8a4d"}
Nov 03 19:54:07.606896 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:07.606192    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:07.760778 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:07.760752    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:07.806423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:07.806391    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-openshift-ca\") pod \"636635fb-6bb9-4b30-bf3f-e3650a20a231\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") "
Nov 03 19:54:07.806613 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:07.806484    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-xks-ca\") pod \"636635fb-6bb9-4b30-bf3f-e3650a20a231\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") "
Nov 03 19:54:07.806708 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:07.806652    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636635fb-6bb9-4b30-bf3f-e3650a20a231-kube-api-access-gn2sw\" (UniqueName: \"kubernetes.io/projected/636635fb-6bb9-4b30-bf3f-e3650a20a231-kube-api-access-gn2sw\") pod \"636635fb-6bb9-4b30-bf3f-e3650a20a231\" (UID: \"636635fb-6bb9-4b30-bf3f-e3650a20a231\") "
Nov 03 19:54:07.806811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:07.806780    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-openshift-ca" pod "636635fb-6bb9-4b30-bf3f-e3650a20a231" (UID: "636635fb-6bb9-4b30-bf3f-e3650a20a231"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:54:07.807523 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:07.807499    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:07.809127 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:07.809099    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-xks-ca" pod "636635fb-6bb9-4b30-bf3f-e3650a20a231" (UID: "636635fb-6bb9-4b30-bf3f-e3650a20a231"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:54:07.809361 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:07.809340    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636635fb-6bb9-4b30-bf3f-e3650a20a231-kube-api-access-gn2sw" pod "636635fb-6bb9-4b30-bf3f-e3650a20a231" (UID: "636635fb-6bb9-4b30-bf3f-e3650a20a231"). InnerVolumeSpecName "kube-api-access-gn2sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:07.909644 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:07.909604    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn2sw\" (UniqueName: \"kubernetes.io/projected/636635fb-6bb9-4b30-bf3f-e3650a20a231-kube-api-access-gn2sw\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:07.909829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:07.909663    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636635fb-6bb9-4b30-bf3f-e3650a20a231-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:08.289709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:08.289663    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:08.491909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:08.491881    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"
Nov 03 19:54:08.491909 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:08.491896    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc" event={"ID":"636635fb-6bb9-4b30-bf3f-e3650a20a231","Type":"ContainerDied","Data":"95400191f3e6be811ddcbe0154868d66876c366d142b6344a5c5e72326a44ff0"}
Nov 03 19:54:08.492132 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:08.491989    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95400191f3e6be811ddcbe0154868d66876c366d142b6344a5c5e72326a44ff0"
Nov 03 19:54:08.689345 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:08.689305    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:09.545205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:09.537766    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:10.299727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:10.299689    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:10.461198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:10.439258    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:11.136413 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:11.136371    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:11.991194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:11.990621    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:12.114633 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:12.114352    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:12.596197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:12.595805    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:12.838478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:12.838439    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:13.413548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:13.413510    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:13.720219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:13.719280    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:13.927937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:13.925085    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:14.494781 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:14.494711    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:14.625632 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:14.625598    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:14.767264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:14.761728    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:15.509817 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:15.495823    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:16.086677 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:16.086643    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:16.355966 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:16.355881    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:16.820054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:16.820011    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:16.962610 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:16.962574    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:17.247206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:17.242795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:18.066365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.066256    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:18.091260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.091228    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"]
Nov 03 19:54:18.092162 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.092144    4531 cpu_manager.go:414] "containerMap: removing container" podUID="636635fb-6bb9-4b30-bf3f-e3650a20a231" containerName="revoke-keys"
Nov 03 19:54:18.092248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.092163    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636635fb-6bb9-4b30-bf3f-e3650a20a231" containerName="revoke-keys"
Nov 03 19:54:18.092404 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.092392    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="636635fb-6bb9-4b30-bf3f-e3650a20a231" containerName="revoke-keys"
Nov 03 19:54:18.096138 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.096115    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:54:18.105105 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.105035    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"]
Nov 03 19:54:18.105748 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.105725    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0d0987\""
Nov 03 19:54:18.106643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.106585    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0d0987-dockercfg-ddthj\""
Nov 03 19:54:18.424110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.207269    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"]
Nov 03 19:54:18.424110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.228210    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:54:18.424110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.352079    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9f9c41-51a6-4c41-9836-d80da3895659-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f\" (UID: \"dc9f9c41-51a6-4c41-9836-d80da3895659\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:54:18.424110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.352293    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22701240-2a28-494b-8dd7-a580ac54e26c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb\" (UID: \"22701240-2a28-494b-8dd7-a580ac54e26c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:54:18.424110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.352392    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpkf9\" (UniqueName: \"kubernetes.io/projected/22701240-2a28-494b-8dd7-a580ac54e26c-kube-api-access-xpkf9\") pod \"payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb\" (UID: \"22701240-2a28-494b-8dd7-a580ac54e26c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:54:18.424110 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.352689    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5n4k\" (UniqueName: \"kubernetes.io/projected/dc9f9c41-51a6-4c41-9836-d80da3895659-kube-api-access-r5n4k\") pod \"payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f\" (UID: \"dc9f9c41-51a6-4c41-9836-d80da3895659\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:54:18.452197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.449690    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:18.637647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.632150    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5n4k\" (UniqueName: \"kubernetes.io/projected/dc9f9c41-51a6-4c41-9836-d80da3895659-kube-api-access-r5n4k\") pod \"payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f\" (UID: \"dc9f9c41-51a6-4c41-9836-d80da3895659\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:54:18.637647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.632420    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9f9c41-51a6-4c41-9836-d80da3895659-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f\" (UID: \"dc9f9c41-51a6-4c41-9836-d80da3895659\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:54:18.653597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.652858    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9f9c41-51a6-4c41-9836-d80da3895659-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f\" (UID: \"dc9f9c41-51a6-4c41-9836-d80da3895659\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:54:18.693145 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.682560    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:18.693145 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.683632    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22701240-2a28-494b-8dd7-a580ac54e26c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb\" (UID: \"22701240-2a28-494b-8dd7-a580ac54e26c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:54:18.693145 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.683782    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpkf9\" (UniqueName: \"kubernetes.io/projected/22701240-2a28-494b-8dd7-a580ac54e26c-kube-api-access-xpkf9\") pod \"payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb\" (UID: \"22701240-2a28-494b-8dd7-a580ac54e26c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:54:18.693145 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.685895    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22701240-2a28-494b-8dd7-a580ac54e26c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb\" (UID: \"22701240-2a28-494b-8dd7-a580ac54e26c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:54:18.769862 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.698196    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"]
Nov 03 19:54:18.838472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.780248    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5n4k\" (UniqueName: \"kubernetes.io/projected/dc9f9c41-51a6-4c41-9836-d80da3895659-kube-api-access-r5n4k\") pod \"payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f\" (UID: \"dc9f9c41-51a6-4c41-9836-d80da3895659\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:54:18.838472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.790008    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpkf9\" (UniqueName: \"kubernetes.io/projected/22701240-2a28-494b-8dd7-a580ac54e26c-kube-api-access-xpkf9\") pod \"payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb\" (UID: \"22701240-2a28-494b-8dd7-a580ac54e26c\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:54:18.891203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:18.885991    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:54:19.032862 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.032811    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:54:19.219073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.219040    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"]
Nov 03 19:54:19.220219 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:54:19.220182    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22701240_2a28_494b_8dd7_a580ac54e26c.slice/crio-1745981a5efa0474addff1324e6cdd65109fe42c558d705c6262155a80e9714b WatchSource:0}: Error finding container 1745981a5efa0474addff1324e6cdd65109fe42c558d705c6262155a80e9714b: Status 404 returned error can't find the container with id 1745981a5efa0474addff1324e6cdd65109fe42c558d705c6262155a80e9714b
Nov 03 19:54:19.391782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.391756    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"]
Nov 03 19:54:19.392353 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:54:19.392324    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9f9c41_51a6_4c41_9836_d80da3895659.slice/crio-2fe5e04e381a34c7d3cf8e58d58089392c7ff96197beeed962ec17a74b0b0e04 WatchSource:0}: Error finding container 2fe5e04e381a34c7d3cf8e58d58089392c7ff96197beeed962ec17a74b0b0e04: Status 404 returned error can't find the container with id 2fe5e04e381a34c7d3cf8e58d58089392c7ff96197beeed962ec17a74b0b0e04
Nov 03 19:54:19.675647 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.675603    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:19.916261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.916223    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f" event={"ID":"dc9f9c41-51a6-4c41-9836-d80da3895659","Type":"ContainerStarted","Data":"0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba"}
Nov 03 19:54:19.916446 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.916298    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f" event={"ID":"dc9f9c41-51a6-4c41-9836-d80da3895659","Type":"ContainerStarted","Data":"2fe5e04e381a34c7d3cf8e58d58089392c7ff96197beeed962ec17a74b0b0e04"}
Nov 03 19:54:19.916446 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.916407    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:54:19.918342 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.918317    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb" event={"ID":"22701240-2a28-494b-8dd7-a580ac54e26c","Type":"ContainerStarted","Data":"237c607765ca1446b416080407030c6d37ffa093810b97918aed7d3ca59009e0"}
Nov 03 19:54:19.918473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.918360    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb" event={"ID":"22701240-2a28-494b-8dd7-a580ac54e26c","Type":"ContainerStarted","Data":"1745981a5efa0474addff1324e6cdd65109fe42c558d705c6262155a80e9714b"}
Nov 03 19:54:19.918473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.918386    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:54:19.933206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.933145    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f" podStartSLOduration=1.933133412 podStartE2EDuration="1.933133412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:19.928904848 +0000 UTC m=+1373.199028142" watchObservedRunningTime="2026-11-03 19:54:19.933133412 +0000 UTC m=+1373.203256706"
Nov 03 19:54:19.964196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:19.963807    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb" podStartSLOduration=1.9637903589999999 podStartE2EDuration="1.963790359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:19.943620128 +0000 UTC m=+1373.213743424" watchObservedRunningTime="2026-11-03 19:54:19.963790359 +0000 UTC m=+1373.233913656"
Nov 03 19:54:20.726893 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:54:20.726822    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:54:21.007893 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:21.007751    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:22.059072 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:22.059025    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:22.463016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:22.462974    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:23.386333 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.383322    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"]
Nov 03 19:54:23.389809 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.389782    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.401792 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.401439    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bdb694\"/\"default-dockercfg-vcqkl\""
Nov 03 19:54:23.401792 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.401552    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bdb694\"/\"test-model-bdb694-kserve-self-signed-certs\""
Nov 03 19:54:23.404255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.403472    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bdb694\"/\"openshift-service-ca.crt\""
Nov 03 19:54:23.404255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.403607    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bdb694\"/\"kube-root-ca.crt\""
Nov 03 19:54:23.409675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.408120    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"]
Nov 03 19:54:23.466782 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.465938    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-dshm\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.473556 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.473517    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-model-cache\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.473722 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.473615    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6t7\" (UniqueName: \"kubernetes.io/projected/abf4257b-2d15-421e-96c0-9650d6f82a57-kube-api-access-nt6t7\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.473722 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.473645    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-tmp-dir\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.473803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.473730    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-home\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.473803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.473744    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf4257b-2d15-421e-96c0-9650d6f82a57-tls-certs\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.575490 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.575453    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-dshm\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.575790 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.575766    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-model-cache\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.575871 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.575847    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6t7\" (UniqueName: \"kubernetes.io/projected/abf4257b-2d15-421e-96c0-9650d6f82a57-kube-api-access-nt6t7\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.575935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.575917    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-tmp-dir\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.576059 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.576044    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-home\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.576112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.576069    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf4257b-2d15-421e-96c0-9650d6f82a57-tls-certs\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.576166 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.576122    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-model-cache\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.580256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.577168    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-tmp-dir\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.580256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.577441    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-home\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.580256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.579462    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-dshm\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.581690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.581666    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf4257b-2d15-421e-96c0-9650d6f82a57-tls-certs\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.583539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.583514    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6t7\" (UniqueName: \"kubernetes.io/projected/abf4257b-2d15-421e-96c0-9650d6f82a57-kube-api-access-nt6t7\") pod \"test-model-bdb694-kserve-98d86d685-7m9ng\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") " pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:23.753829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:23.752317    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:24.164800 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:54:24.164766    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabf4257b_2d15_421e_96c0_9650d6f82a57.slice/crio-7bfce0782243b943174e2c31dbff6b99cad262deaa2a8edef7eb13f97f315055 WatchSource:0}: Error finding container 7bfce0782243b943174e2c31dbff6b99cad262deaa2a8edef7eb13f97f315055: Status 404 returned error can't find the container with id 7bfce0782243b943174e2c31dbff6b99cad262deaa2a8edef7eb13f97f315055
Nov 03 19:54:24.168475 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:24.168437    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"]
Nov 03 19:54:24.971877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:24.971833    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" event={"ID":"abf4257b-2d15-421e-96c0-9650d6f82a57","Type":"ContainerStarted","Data":"7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691"}
Nov 03 19:54:24.972409 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:24.971893    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" event={"ID":"abf4257b-2d15-421e-96c0-9650d6f82a57","Type":"ContainerStarted","Data":"7bfce0782243b943174e2c31dbff6b99cad262deaa2a8edef7eb13f97f315055"}
Nov 03 19:54:24.995543 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:24.995482    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" podStartSLOduration=1.995468536 podStartE2EDuration="1.995468536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:24.992564493 +0000 UTC m=+1378.262687786" watchObservedRunningTime="2026-11-03 19:54:24.995468536 +0000 UTC m=+1378.265591830"
Nov 03 19:54:25.116283 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:25.116241    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:25.273251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:25.268443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:25.759577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:25.759549    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:26.219198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:26.213793    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:26.909837 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:26.909801    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:27.059925 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:27.059886    4531 generic.go:356] "Generic (PLEG): container finished" podID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerID="d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168" exitCode=0
Nov 03 19:54:27.060483 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:27.060454    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" event={"ID":"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d","Type":"ContainerDied","Data":"d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168"}
Nov 03 19:54:27.060934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:27.060914    4531 scope.go:122] "RemoveContainer" containerID="d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168"
Nov 03 19:54:27.063654 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:27.063620    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:27.063774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:27.063705    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:27.063774 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:27.063756    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:27.647636 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:27.647594    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:28.075861 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:28.075768    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" event={"ID":"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d","Type":"ContainerStarted","Data":"093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3"}
Nov 03 19:54:28.262238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:28.261728    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:54:28.263330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:28.262718    4531 patch_prober.go:29] interesting pod/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 19:54:28.263330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:28.262806    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 19:54:28.511075 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:28.511032    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:29.074963 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:29.074924    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:29.226123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:29.226085    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:29.261200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:29.257076    4531 patch_prober.go:29] interesting pod/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 19:54:29.261200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:29.257137    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 19:54:29.541197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:29.534294    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:30.252193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:30.249588    4531 patch_prober.go:29] interesting pod/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 19:54:30.252193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:30.249685    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 19:54:30.420194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:30.417986    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:30.615246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:30.614733    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:31.159232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:31.131911    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:54:31.262360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:31.259988    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:54:31.262360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:31.260034    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:54:31.262360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:31.232502    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:31.262360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:31.258532    4531 patch_prober.go:29] interesting pod/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 19:54:31.262360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:31.260380    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 19:54:31.605293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:31.591670    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:31.637325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:31.637284    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:54:32.254198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:32.250233    4531 patch_prober.go:29] interesting pod/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 19:54:32.254198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:32.250296    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 19:54:32.874885 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:32.874848    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:33.302207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:33.278114    4531 patch_prober.go:29] interesting pod/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 19:54:33.302207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:33.278193    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 19:54:33.657796 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:33.657759    4531 scope.go:122] "RemoveContainer" containerID="9fd50fe798186a802b97ec9c73d1445536fb1f5e97f871456fe49f6ede2aed6a"
Nov 03 19:54:33.829787 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:33.829751    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:33.871240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:33.871147    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:54:33.952263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:33.944436    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:34.059233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:34.053904    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:54:34.199292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:34.199244    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:54:34.224560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:34.224474    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:34.265199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:34.248925    4531 patch_prober.go:29] interesting pod/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 19:54:34.265199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:34.249096    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 19:54:34.641246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:34.639210    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:34.839202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:34.832848    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:35.236203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:35.233209    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:35.291505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:35.291468    4531 patch_prober.go:29] interesting pod/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 19:54:35.291788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:35.291755    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 19:54:35.749051 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:35.637372    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:35.763286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:35.752122    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:54:35.831204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:35.770258    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:54:35.846218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:35.834022    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Nov 03 19:54:36.270311 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:35.887775    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:36.393360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.084364    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:36.413867 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.118201    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:54:36.415501 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.118246    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:54:36.430485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.401269    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Nov 03 19:54:36.430723 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.267981    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:54:36.432021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.407915    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:36.459398 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.267535    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:54:36.462681 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.462260    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"]
Nov 03 19:54:36.648489 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.648456    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:36.660222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.657125    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:36.703740 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.703700    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"]
Nov 03 19:54:36.704084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.704062    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:54:36.704372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.704353    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:54:36.724463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.724428    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:36.745644 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.745612    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:54:36.748001 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.745789    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:54:36.769563 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.746305    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-332694\"/\"default-dockercfg-shkdw\""
Nov 03 19:54:36.796115 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.746431    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-332694\"/\"test-model-332694-kserve-self-signed-certs\""
Nov 03 19:54:36.796115 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.770138    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:54:36.796115 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.746534    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-332694\"/\"openshift-service-ca.crt\""
Nov 03 19:54:36.829951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.746628    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-332694\"/\"kube-root-ca.crt\""
Nov 03 19:54:36.829951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.804461    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:36.829951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.813315    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:54:36.829951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.813366    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v" podUID="c6beddf2-5be3-42ae-a6c7-6256834bd41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:36.829951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.825833    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:36.855197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.844633    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tls-certs\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:36.855197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.844777    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-home\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:36.855197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.844920    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbp29\" (UniqueName: \"kubernetes.io/projected/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-kube-api-access-zbp29\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:36.855197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.845100    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-model-cache\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:36.855197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.845270    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tmp-dir\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:36.855197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.845430    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-dshm\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:36.889340 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.860198    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:54:37.026253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.889592    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:54:37.026253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:36.988571    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:54:37.171227 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.171187    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tls-certs\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.171546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.171519    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-home\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.171751 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.171730    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbp29\" (UniqueName: \"kubernetes.io/projected/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-kube-api-access-zbp29\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.171992 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.171973    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-model-cache\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.172228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.172208    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tmp-dir\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.172470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.172450    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-dshm\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.190685 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.190650    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-model-cache\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.218198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.194559    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-home\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.316357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.294774    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tmp-dir\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.339374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.339339    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbp29\" (UniqueName: \"kubernetes.io/projected/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-kube-api-access-zbp29\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.366196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.351014    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tls-certs\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.450199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.425704    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:37.495833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.431532    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:37.639383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.431599    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:37.639383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.431655    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:37.639383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.496521    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-dshm\") pod \"test-model-332694-kserve-865b9db95c-cjwdk\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") " pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.639383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.501469    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:54:37.701249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.674912    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:37.878984 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.877925    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:37.878984 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.845986    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": context deadline exceeded"
Nov 03 19:54:37.880155 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.872676    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:54:37.881283 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.872697    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:37.980214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:37.872628    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:38.087205 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:54:38.084065    4531 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.094s"
Nov 03 19:54:38.094420 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.094388    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:54:38.094623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.094610    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:54:38.094784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.094771    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:54:38.094898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.094888    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:54:38.095011 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.095002    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:54:38.095112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.095103    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:54:38.095255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.095245    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:38.095367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.095358    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:54:38.322279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.322135    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:54:38.355925 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.325684    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:38.356329 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.095512    4531 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:54:38.518330 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.518287    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:38.540240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.522660    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:38.548854 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.548815    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:38.551012 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.550987    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:54:38.581835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.581322    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:38.585711 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.585274    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:38.774339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.774280    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:38.775348 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.775301    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:38.775939 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.775916    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:54:38.776741 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.776716    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:54:38.880253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.852799    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:54:38.880253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.852861    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 19:54:38.925316 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.925269    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:38.942292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.933294    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:38.962763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.941610    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:38.987848 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:38.987798    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:54:39.028426 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.028378    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 19:54:39.075921 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.074558    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:54:39.075921 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.074621    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:54:39.075921 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.074658    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:54:39.117639 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.117606    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 19:54:39.132280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.132159    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:54:39.132280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.132256    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:54:39.136232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.136201    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 19:54:39.176379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.176348    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:54:39.179024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.178999    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:54:39.425368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.425336    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"]
Nov 03 19:54:39.688842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:39.688757    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:40.125205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:40.125076    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" event={"ID":"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46","Type":"ContainerStarted","Data":"40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574"}
Nov 03 19:54:40.125205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:40.125130    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" event={"ID":"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46","Type":"ContainerStarted","Data":"4643731c97aa1cd3a3c63c808f23a1205e346b7baed2e490280f4ccdae109848"}
Nov 03 19:54:40.156207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:40.143990    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" podStartSLOduration=5.143970954 podStartE2EDuration="5.143970954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:40.14372849 +0000 UTC m=+1393.413851786" watchObservedRunningTime="2026-11-03 19:54:40.143970954 +0000 UTC m=+1393.414094251"
Nov 03 19:54:40.923287 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:40.923248    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:41.009265 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:41.009226    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:41.164702 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:41.164664    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:41.292792 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:41.292702    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:42.074234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.056682    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:42.262245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.258230    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"]
Nov 03 19:54:42.367122 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.367035    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:42.370468 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.370442    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.386203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.385771    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ffe132\"/\"kube-root-ca.crt\""
Nov 03 19:54:42.386203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.385912    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ffe132\"/\"openshift-service-ca.crt\""
Nov 03 19:54:42.390251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.386702    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ffe132\"/\"default-dockercfg-76qwn\""
Nov 03 19:54:42.390251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.386861    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ffe132\"/\"test-model-ffe132-kserve-self-signed-certs\""
Nov 03 19:54:42.402734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.401475    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"]
Nov 03 19:54:42.563296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.538130    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-dshm\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.563296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.538317    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-home\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.563296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.543104    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-model-cache\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.563296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.543330    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd542f86-cf28-4141-b7f7-b0661226c455-tls-certs\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.563296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.543451    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-tmp-dir\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.563296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.543653    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhqs\" (UniqueName: \"kubernetes.io/projected/cd542f86-cf28-4141-b7f7-b0661226c455-kube-api-access-vbhqs\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.666314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.652123    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-dshm\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.666314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.652321    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-home\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.666314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.652382    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-model-cache\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.666314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.652502    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd542f86-cf28-4141-b7f7-b0661226c455-tls-certs\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.666314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.652593    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-tmp-dir\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.666314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.652755    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhqs\" (UniqueName: \"kubernetes.io/projected/cd542f86-cf28-4141-b7f7-b0661226c455-kube-api-access-vbhqs\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.666314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.655797    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-tmp-dir\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.666314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.656156    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-home\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.666314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.656394    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-model-cache\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.666314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.661497    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-dshm\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.679067 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.679027    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd542f86-cf28-4141-b7f7-b0661226c455-tls-certs\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.687878 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.687839    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhqs\" (UniqueName: \"kubernetes.io/projected/cd542f86-cf28-4141-b7f7-b0661226c455-kube-api-access-vbhqs\") pod \"test-model-ffe132-kserve-5cb4d484f7-sjgxl\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") " pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:42.904013 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:42.903968    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:43.165290 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:43.160794    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:43.251355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:43.251322    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"]
Nov 03 19:54:43.252984 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:54:43.252949    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd542f86_cf28_4141_b7f7_b0661226c455.slice/crio-8b93d99d0c15da2442a3e352652c70a483bb2eedb0dcb6aa268a865c81ca4d29 WatchSource:0}: Error finding container 8b93d99d0c15da2442a3e352652c70a483bb2eedb0dcb6aa268a865c81ca4d29: Status 404 returned error can't find the container with id 8b93d99d0c15da2442a3e352652c70a483bb2eedb0dcb6aa268a865c81ca4d29
Nov 03 19:54:43.393421 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:43.393376    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl" event={"ID":"cd542f86-cf28-4141-b7f7-b0661226c455","Type":"ContainerStarted","Data":"3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79"}
Nov 03 19:54:43.393603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:43.393440    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl" event={"ID":"cd542f86-cf28-4141-b7f7-b0661226c455","Type":"ContainerStarted","Data":"8b93d99d0c15da2442a3e352652c70a483bb2eedb0dcb6aa268a865c81ca4d29"}
Nov 03 19:54:43.410714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:43.410655    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl" podStartSLOduration=1.4106389799999999 podStartE2EDuration="1.41063898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:43.407154861 +0000 UTC m=+1396.677278156" watchObservedRunningTime="2026-11-03 19:54:43.41063898 +0000 UTC m=+1396.680762310"
Nov 03 19:54:43.904935 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:43.904905    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:43.933306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:43.933267    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:44.824412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:44.824376    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:45.546232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:45.546138    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:46.219618 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:46.219570    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:46.396193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:46.396038    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:46.661203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:46.654433    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:46.740194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:46.740026    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"]
Nov 03 19:54:46.756196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:46.754733    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a9cd5e-cjxgc"]
Nov 03 19:54:46.870198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:46.869492    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636635fb-6bb9-4b30-bf3f-e3650a20a231" path="/var/lib/kubelet/pods/636635fb-6bb9-4b30-bf3f-e3650a20a231/volumes"
Nov 03 19:54:46.873918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:46.873853    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:47.204485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.204450    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:47.788246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.779321    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:47.831266 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.826519    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"]
Nov 03 19:54:47.835518 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.833472    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:47.843918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.841980    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"]
Nov 03 19:54:47.904134 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.904104    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-332694-openshift-default-dockercfg-pvj4g\""
Nov 03 19:54:47.919428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.918413    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:47.919428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.918474    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:47.919428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.918562    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30571081-5efa-4101-9cb2-06cd0be6d07c-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:47.919428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.918683    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:47.919428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.919311    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:47.919428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.919367    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:47.920038 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.919998    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbqjk\" (UniqueName: \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-kube-api-access-dbqjk\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:47.920160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.920113    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:47.920281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.920222    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:47.923167 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.923142    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:47.939740 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:47.939712    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:48.021961 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.021924    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.022185 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.021985    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.022185 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.022148    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbqjk\" (UniqueName: \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-kube-api-access-dbqjk\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.022353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.022303    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.022430 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.022345    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.022430 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.022385    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.022549 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.022462    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.022608 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.022594    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.022738 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.022615    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.022738 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.022660    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30571081-5efa-4101-9cb2-06cd0be6d07c-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.022839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.022740    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.022839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.022754    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.023081 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.023056    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.023372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.023355    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30571081-5efa-4101-9cb2-06cd0be6d07c-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.024995 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.024966    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.025235 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.025218    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.028766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.028742    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.028976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.028958    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbqjk\" (UniqueName: \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-kube-api-access-dbqjk\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.182764 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.182726    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:48.467978 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.467881    4531 generic.go:356] "Generic (PLEG): container finished" podID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerID="282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5" exitCode=0
Nov 03 19:54:48.468146 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.468047    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" event={"ID":"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754","Type":"ContainerDied","Data":"282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5"}
Nov 03 19:54:48.473553 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.473487    4531 scope.go:122] "RemoveContainer" containerID="282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5"
Nov 03 19:54:48.478198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.478090    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:48.478323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.478252    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:48.478393 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.478343    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:48.482716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.482689    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"]
Nov 03 19:54:48.489842 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.489821    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:54:48.503078 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.503037    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:48.503237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.503123    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:48.503237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.503167    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:48.683201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:48.683127    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:49.490258 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:49.490221    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" event={"ID":"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754","Type":"ContainerStarted","Data":"198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e"}
Nov 03 19:54:49.496025 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:49.495985    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" event={"ID":"30571081-5efa-4101-9cb2-06cd0be6d07c","Type":"ContainerStarted","Data":"cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b"}
Nov 03 19:54:49.496243 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:49.496076    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" event={"ID":"30571081-5efa-4101-9cb2-06cd0be6d07c","Type":"ContainerStarted","Data":"d1824c74d63e4739c66852e54e34439184ab628513823506a926a14f95f71cff"}
Nov 03 19:54:49.530356 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:49.530300    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" podStartSLOduration=2.530286774 podStartE2EDuration="2.530286774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:49.526000427 +0000 UTC m=+1402.796123723" watchObservedRunningTime="2026-11-03 19:54:49.530286774 +0000 UTC m=+1402.800410068"
Nov 03 19:54:49.567199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:49.567105    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:49.868256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:49.867567    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:54:49.868256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:49.867679    4531 patch_prober.go:29] interesting pod/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 19:54:49.868256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:49.867724    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 19:54:50.185571 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:50.185534    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:50.189343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:50.189304    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 19:54:50.189597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:50.189573    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 19:54:50.618249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:50.580957    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:50.809916 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:54:50.809873    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:54:50.877260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:50.876913    4531 patch_prober.go:29] interesting pod/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 19:54:50.877260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:50.876971    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 19:54:51.185949 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:51.183549    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 19:54:51.185949 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:51.183615    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 19:54:51.876202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:51.875331    4531 patch_prober.go:29] interesting pod/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 19:54:51.876202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:51.875392    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 19:54:51.894197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:51.891062    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:52.186199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:52.185550    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 19:54:52.186199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:52.185619    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 19:54:52.890847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:52.890493    4531 patch_prober.go:29] interesting pod/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 19:54:52.890847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:52.890554    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 19:54:52.904338 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:52.904305    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:52.920657 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:52.920616    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:53.197133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.197038    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 19:54:53.197406 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.197380    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 19:54:53.203823 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.203790    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:53.479253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.478769    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"]
Nov 03 19:54:53.513695 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.513610    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:53.518581 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.518552    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.530726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.526188    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a99fad\"/\"default-dockercfg-ggkt8\""
Nov 03 19:54:53.530726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.526337    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a99fad\"/\"test-model-a99fad-kserve-self-signed-certs\""
Nov 03 19:54:53.530726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.527032    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a99fad\"/\"openshift-service-ca.crt\""
Nov 03 19:54:53.530726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.527159    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a99fad\"/\"kube-root-ca.crt\""
Nov 03 19:54:53.530726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.530101    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-dshm\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.530726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.530202    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5czz\" (UniqueName: \"kubernetes.io/projected/eede0436-4013-49f2-bbf6-00dade948787-kube-api-access-t5czz\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.530726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.530243    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-model-cache\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.530726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.530284    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-home\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.530726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.530372    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eede0436-4013-49f2-bbf6-00dade948787-tls-certs\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.530726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.530420    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-tmp-dir\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.532280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.532257    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"]
Nov 03 19:54:53.620427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.620394    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 19:54:53.634417 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.634381    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eede0436-4013-49f2-bbf6-00dade948787-tls-certs\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.634606 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.634497    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-tmp-dir\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.634780 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.634752    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-dshm\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.634937 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.634889    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5czz\" (UniqueName: \"kubernetes.io/projected/eede0436-4013-49f2-bbf6-00dade948787-kube-api-access-t5czz\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.635021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.634943    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-model-cache\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.635021 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.635009    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-home\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.635701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.635673    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-home\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.722234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.708206    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:53.722234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.708764    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-model-cache\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:53.722234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:53.709160    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-tmp-dir\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:54.125932 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.124820    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eede0436-4013-49f2-bbf6-00dade948787-tls-certs\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:54.146245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.141586    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-dshm\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:54.226232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.225061    4531 patch_prober.go:29] interesting pod/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 19:54:54.226232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.225207    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 19:54:54.329460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.303020    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:54.422297 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.363738    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 19:54:54.466246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.450471    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 19:54:54.499211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.392879    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"]
Nov 03 19:54:54.499211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.486204    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy" containerID="cri-o://093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3" gracePeriod=30
Nov 03 19:54:54.509550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.125508    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5czz\" (UniqueName: \"kubernetes.io/projected/eede0436-4013-49f2-bbf6-00dade948787-kube-api-access-t5czz\") pod \"test-model-a99fad-kserve-7f8946b455-hmv9b\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") " pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:54.689277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.680686    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:54.805301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.805266    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:54.892201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.871686    4531 patch_prober.go:29] interesting pod/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 19:54:54.904582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.904539    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 19:54:54.910641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:54.910618    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:54:55.106587 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:55.106546    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:55.315088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:55.294454    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 19:54:55.315088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:55.294531    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 19:54:55.393196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:55.391415    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:55.573254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:55.569676    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"]
Nov 03 19:54:55.916907 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:55.915778    4531 patch_prober.go:29] interesting pod/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 19:54:55.916907 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:55.915846    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 19:54:55.924653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:55.924615    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:56.183206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:56.183100    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 19:54:56.183462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:56.183431    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 19:54:56.495441 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:56.495358    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:56.645485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:56.640633    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:56.769210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:56.769097    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:56.871716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:56.871673    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:54:56.873514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:56.873492    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:54:56.948824 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:56.948785    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" event={"ID":"eede0436-4013-49f2-bbf6-00dade948787","Type":"ContainerStarted","Data":"07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644"}
Nov 03 19:54:56.949029 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:56.948858    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" event={"ID":"eede0436-4013-49f2-bbf6-00dade948787","Type":"ContainerStarted","Data":"7ed800307495b3ff5afece077a410a073bfe9231b5b3707f1735a43b39233e0c"}
Nov 03 19:54:56.965073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:56.965018    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" podStartSLOduration=3.965005882 podStartE2EDuration="3.965005882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:56.964833951 +0000 UTC m=+1410.234957249" watchObservedRunningTime="2026-11-03 19:54:56.965005882 +0000 UTC m=+1410.235129240"
Nov 03 19:54:57.320333 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:57.314526    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:57.320333 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:57.316547    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:57.323302 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:57.323277    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 19:54:57.602590 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:57.601084    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:57.805979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:57.805952    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:54:58.421197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:58.419766    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:59.123197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:59.118969    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:59.824981 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:59.824955    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:54:59.927724 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:59.924336    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:59.987238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:59.981336    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-credential-socket\") pod \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") "
Nov 03 19:55:00.004653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:54:59.999270    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-credential-socket" pod "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" (UID: "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:55:00.004653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.000032    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-token\" (UniqueName: \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-token\") pod \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") "
Nov 03 19:55:00.004653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.000120    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-envoy\") pod \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") "
Nov 03 19:55:00.004653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.000183    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istiod-ca-cert\") pod \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") "
Nov 03 19:55:00.004653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.001031    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istiod-ca-cert" pod "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" (UID: "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:00.004653 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.004376    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-kube-api-access-f4q8v\" (UniqueName: \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-kube-api-access-f4q8v\") pod \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") "
Nov 03 19:55:00.009194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.005364    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-token" pod "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" (UID: "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:00.009194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.008754    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-socket\") pod \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") "
Nov 03 19:55:00.009194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.009012    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-socket" pod "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" (UID: "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:55:00.009466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.009300    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-certs" pod "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" (UID: "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:55:00.015761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.011106    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-certs\") pod \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") "
Nov 03 19:55:00.015761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.011193    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-data\") pod \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") "
Nov 03 19:55:00.015761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.011269    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-podinfo\") pod \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\" (UID: \"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d\") "
Nov 03 19:55:00.015761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.011335    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-kube-api-access-f4q8v" pod "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" (UID: "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d"). InnerVolumeSpecName "kube-api-access-f4q8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:00.015761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.011466    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-data" pod "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" (UID: "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:55:00.015761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.012108    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-envoy" pod "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" (UID: "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:55:00.015761 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.013723    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-podinfo" pod "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" (UID: "9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:55:00.019009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.018897    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.019009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.018918    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.019009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.018928    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.019009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.018938    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.019009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.018949    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.019009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.018957    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.019009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.018967    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.019009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.018978    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4q8v\" (UniqueName: \"kubernetes.io/projected/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-kube-api-access-f4q8v\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.019009 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.018987    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.024517 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.024488    4531 generic.go:356] "Generic (PLEG): container finished" podID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerID="093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3" exitCode=0
Nov 03 19:55:00.024632 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.024544    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"
Nov 03 19:55:00.024632 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.024570    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" event={"ID":"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d","Type":"ContainerDied","Data":"093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3"}
Nov 03 19:55:00.024717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.024637    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb" event={"ID":"9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d","Type":"ContainerDied","Data":"bd4ee1ff50953afebbfa450fdb7df6311ee5a10647b81d83d54752d5b51fde92"}
Nov 03 19:55:00.024717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.024670    4531 scope.go:122] "RemoveContainer" containerID="093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3"
Nov 03 19:55:00.057530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.057489    4531 scope.go:122] "RemoveContainer" containerID="d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168"
Nov 03 19:55:00.072787 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.072751    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"]
Nov 03 19:55:00.076375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.076350    4531 scope.go:122] "RemoveContainer" containerID="093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3"
Nov 03 19:55:00.076855 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:55:00.076829    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3\": container with ID starting with 093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3 not found: ID does not exist" containerID="093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3"
Nov 03 19:55:00.076952 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.076862    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3"} err="failed to get container status \"093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3\": rpc error: code = NotFound desc = could not find container \"093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3\": container with ID starting with 093cf32e0a09e4a59571e291ddca54be244ba8eac652cb51a9f6a41da1af18d3 not found: ID does not exist"
Nov 03 19:55:00.076952 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.076881    4531 scope.go:122] "RemoveContainer" containerID="d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168"
Nov 03 19:55:00.077149 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:55:00.077132    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168\": container with ID starting with d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168 not found: ID does not exist" containerID="d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168"
Nov 03 19:55:00.077245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.077153    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168"} err="failed to get container status \"d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168\": rpc error: code = NotFound desc = could not find container \"d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168\": container with ID starting with d591cb2292e71444d66150592293522a9c3ec4d27c82da57ca678dedd8da8168 not found: ID does not exist"
Nov 03 19:55:00.078614 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.078594    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a9cd5e-openshift-default-7c6b7fc5fc-zchpb"]
Nov 03 19:55:00.307243 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.307114    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:00.447494 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.447455    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:00.863749 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:00.863714    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" path="/var/lib/kubelet/pods/9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d/volumes"
Nov 03 19:55:01.078371 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:01.078337    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:01.539151 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:01.539089    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:01.738195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:01.737760    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:02.323215 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:02.319758    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:03.271533 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:03.261383    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:04.019850 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:04.018222    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:04.721150 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:04.720870    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:04.816196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:04.815709    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:55:04.832517 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:04.832488    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:55:05.153715 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:05.153680    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:55:05.655533 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:05.655502    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:06.553607 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:06.553569    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:07.183425 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:07.183368    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:07.956089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:07.956047    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:08.105070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.105035    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:08.697443 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.697401    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"]
Nov 03 19:55:08.698454 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.698437    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy"
Nov 03 19:55:08.698521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.698456    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy"
Nov 03 19:55:08.698521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.698478    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy"
Nov 03 19:55:08.698521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.698483    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy"
Nov 03 19:55:08.698625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.698614    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy"
Nov 03 19:55:08.698660 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.698625    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a76bcd7-fe2a-43cd-8a21-4baf4a8f793d" containerName="istio-proxy"
Nov 03 19:55:08.704353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.704326    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:08.712833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.712805    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"]
Nov 03 19:55:08.731532 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.731504    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:55:08.731676 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.731547    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:55:08.793056 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.793020    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:08.873658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.873599    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c8eec-t699p\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:08.911450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.911410    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c8eec-t699p\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:08.912253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:08.912221    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p496d\" (UniqueName: \"kubernetes.io/projected/7758898d-f2e3-4d18-af29-34326ffae000-kube-api-access-p496d\") pod \"maas-api-revoke-keys-e2e-api-a-0c8eec-t699p\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:09.034275 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:09.034161    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p496d\" (UniqueName: \"kubernetes.io/projected/7758898d-f2e3-4d18-af29-34326ffae000-kube-api-access-p496d\") pod \"maas-api-revoke-keys-e2e-api-a-0c8eec-t699p\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:09.034730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:09.034447    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c8eec-t699p\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:09.034730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:09.034530    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c8eec-t699p\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:09.035094 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:09.035071    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c8eec-t699p\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:09.037050 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:09.037030    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c8eec-t699p\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:09.041015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:09.040992    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p496d\" (UniqueName: \"kubernetes.io/projected/7758898d-f2e3-4d18-af29-34326ffae000-kube-api-access-p496d\") pod \"maas-api-revoke-keys-e2e-api-a-0c8eec-t699p\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:09.326372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:09.326289    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:09.601371 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:09.599650    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:09.632221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:09.632190    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"]
Nov 03 19:55:09.634660 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:55:09.634634    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7758898d_f2e3_4d18_af29_34326ffae000.slice/crio-ccf5ac284312fcedf81c949a535bf82ee8d0e7a398b99624ed59eb7b9a28a336 WatchSource:0}: Error finding container ccf5ac284312fcedf81c949a535bf82ee8d0e7a398b99624ed59eb7b9a28a336: Status 404 returned error can't find the container with id ccf5ac284312fcedf81c949a535bf82ee8d0e7a398b99624ed59eb7b9a28a336
Nov 03 19:55:10.221548 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:10.221504    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:10.237547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:10.237503    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p" event={"ID":"7758898d-f2e3-4d18-af29-34326ffae000","Type":"ContainerStarted","Data":"526e66124d30ea0f79ba8d74fbd744387b6c6b4830c441659290fb876e458fa4"}
Nov 03 19:55:10.237737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:10.237579    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p" event={"ID":"7758898d-f2e3-4d18-af29-34326ffae000","Type":"ContainerStarted","Data":"ccf5ac284312fcedf81c949a535bf82ee8d0e7a398b99624ed59eb7b9a28a336"}
Nov 03 19:55:10.267635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:10.267572    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p" podStartSLOduration=2.267553753 podStartE2EDuration="2.267553753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:10.259060961 +0000 UTC m=+1423.529184257" watchObservedRunningTime="2026-11-03 19:55:10.267553753 +0000 UTC m=+1423.537677047"
Nov 03 19:55:11.070256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:11.069908    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:11.247992 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:11.247955    4531 generic.go:356] "Generic (PLEG): container finished" podID="7758898d-f2e3-4d18-af29-34326ffae000" containerID="526e66124d30ea0f79ba8d74fbd744387b6c6b4830c441659290fb876e458fa4" exitCode=0
Nov 03 19:55:11.248419 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:11.248036    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p" event={"ID":"7758898d-f2e3-4d18-af29-34326ffae000","Type":"ContainerDied","Data":"526e66124d30ea0f79ba8d74fbd744387b6c6b4830c441659290fb876e458fa4"}
Nov 03 19:55:11.733201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:11.727249    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:12.314372 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:12.314332    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:12.532262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:12.532202    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:12.622801 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:12.622762    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-openshift-ca\") pod \"7758898d-f2e3-4d18-af29-34326ffae000\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") "
Nov 03 19:55:12.623059 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:12.623042    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-xks-ca\") pod \"7758898d-f2e3-4d18-af29-34326ffae000\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") "
Nov 03 19:55:12.623106 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:12.623094    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7758898d-f2e3-4d18-af29-34326ffae000-kube-api-access-p496d\" (UniqueName: \"kubernetes.io/projected/7758898d-f2e3-4d18-af29-34326ffae000-kube-api-access-p496d\") pod \"7758898d-f2e3-4d18-af29-34326ffae000\" (UID: \"7758898d-f2e3-4d18-af29-34326ffae000\") "
Nov 03 19:55:12.623149 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:12.623111    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-openshift-ca" pod "7758898d-f2e3-4d18-af29-34326ffae000" (UID: "7758898d-f2e3-4d18-af29-34326ffae000"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:12.623785 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:12.623758    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:12.625188 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:12.625145    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-xks-ca" pod "7758898d-f2e3-4d18-af29-34326ffae000" (UID: "7758898d-f2e3-4d18-af29-34326ffae000"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:55:12.625314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:12.625290    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7758898d-f2e3-4d18-af29-34326ffae000-kube-api-access-p496d" pod "7758898d-f2e3-4d18-af29-34326ffae000" (UID: "7758898d-f2e3-4d18-af29-34326ffae000"). InnerVolumeSpecName "kube-api-access-p496d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:12.725969 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:12.725923    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7758898d-f2e3-4d18-af29-34326ffae000-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:12.725969 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:12.725962    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p496d\" (UniqueName: \"kubernetes.io/projected/7758898d-f2e3-4d18-af29-34326ffae000-kube-api-access-p496d\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:13.058303 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:13.058264    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:13.266389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:13.266359    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"
Nov 03 19:55:13.266577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:13.266396    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p" event={"ID":"7758898d-f2e3-4d18-af29-34326ffae000","Type":"ContainerDied","Data":"ccf5ac284312fcedf81c949a535bf82ee8d0e7a398b99624ed59eb7b9a28a336"}
Nov 03 19:55:13.266577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:13.266448    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccf5ac284312fcedf81c949a535bf82ee8d0e7a398b99624ed59eb7b9a28a336"
Nov 03 19:55:13.839295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:13.828871    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:14.457325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:14.457284    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:15.181755 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:15.180330    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:16.118458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:16.118420    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:16.259389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:16.259349    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:17.239362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:17.230757    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:17.933931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:17.933889    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:18.684135 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:18.681344    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:18.792268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:18.792230    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"]
Nov 03 19:55:18.803328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:18.803299    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"]
Nov 03 19:55:18.841385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:18.841350    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:18.973776 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:18.973696    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:19.758808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:19.758713    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:20.105105 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:20.105012    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:20.577776 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:55:20.577741    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:55:20.577945 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:20.577838    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:20.778789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:20.778752    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:20.946913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:20.946875    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"]
Nov 03 19:55:20.949161 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:20.949130    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7758898d-f2e3-4d18-af29-34326ffae000" containerName="revoke-keys"
Nov 03 19:55:20.949363 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:20.949345    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7758898d-f2e3-4d18-af29-34326ffae000" containerName="revoke-keys"
Nov 03 19:55:20.949716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:20.949698    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7758898d-f2e3-4d18-af29-34326ffae000" containerName="revoke-keys"
Nov 03 19:55:20.955549 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:20.955530    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:20.968106 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:20.965102    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-79c47f-openshift-default-dockercfg-ftrfj\""
Nov 03 19:55:20.978747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:20.978714    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"]
Nov 03 19:55:21.024957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.024905    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:21.138198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.125492    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-data\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.140304 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.140025    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-podinfo\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.146702 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.146670    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-credential-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.147219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.147094    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-token\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.147496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.147391    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istiod-ca-cert\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.147893 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.147872    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.179195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.148057    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqsc\" (UniqueName: \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-kube-api-access-wkqsc\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.179195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.171609    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:21.179195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.178525    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-envoy\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.179195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.178670    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-certs\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.280716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.280611    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-podinfo\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.282605 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.282577    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-credential-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.282832 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.282813    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-token\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.282977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.282962    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istiod-ca-cert\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.283215 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.283168    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-credential-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.288320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.288291    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.288520 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.288501    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqsc\" (UniqueName: \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-kube-api-access-wkqsc\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.288631 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.288617    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-envoy\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.288816 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.288799    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-certs\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.289044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.289028    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-data\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.299576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.289794    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-data\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.299772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.296559    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.299877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.297301    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istiod-ca-cert\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.299975 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.299268    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-certs\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.309198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.305634    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-envoy\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.322205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.320939    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqsc\" (UniqueName: \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-kube-api-access-wkqsc\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.332194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.328094    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-token\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.340880 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.340776    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-podinfo\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.603542 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.603457    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:21.878287 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.876885    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:21.896713 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:55:21.896679    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6ef0ea_9769_4121_8f44_39adf7de2e3b.slice/crio-e337cc1d0d4b949d97b791b05ecb74e6aef896af80d363521a347cb674e3b3ad WatchSource:0}: Error finding container e337cc1d0d4b949d97b791b05ecb74e6aef896af80d363521a347cb674e3b3ad: Status 404 returned error can't find the container with id e337cc1d0d4b949d97b791b05ecb74e6aef896af80d363521a347cb674e3b3ad
Nov 03 19:55:21.897625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.897600    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"]
Nov 03 19:55:21.899800 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.899767    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:21.899884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.899834    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:21.899884 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:21.899864    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:22.268765 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:22.268729    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:22.467200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:22.459266    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:22.480841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:22.480796    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" event={"ID":"bd6ef0ea-9769-4121-8f44-39adf7de2e3b","Type":"ContainerStarted","Data":"3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614"}
Nov 03 19:55:22.481144 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:22.481117    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" event={"ID":"bd6ef0ea-9769-4121-8f44-39adf7de2e3b","Type":"ContainerStarted","Data":"e337cc1d0d4b949d97b791b05ecb74e6aef896af80d363521a347cb674e3b3ad"}
Nov 03 19:55:22.520294 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:22.520154    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" podStartSLOduration=2.520132742 podStartE2EDuration="2.520132742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:22.519451648 +0000 UTC m=+1435.789574951" watchObservedRunningTime="2026-11-03 19:55:22.520132742 +0000 UTC m=+1435.790256039"
Nov 03 19:55:22.603836 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:22.603775    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:22.611301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:22.611276    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:22.708817 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:22.708781    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:23.362004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:23.357078    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:23.490460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:23.490419    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:23.491508 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:23.491483    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:55:23.567101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:23.567065    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:23.792762 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:23.792717    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt" podUID="a527b3f0-9f65-4d15-b849-432b1bb3e445" containerName="payload-processing" containerID="cri-o://6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251" gracePeriod=25
Nov 03 19:55:23.804259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:23.804221    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8" podUID="7a5555cb-2681-4a70-af23-ba1281bddc74" containerName="payload-pre-processing" containerID="cri-o://3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37" gracePeriod=25
Nov 03 19:55:24.273194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.256442    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:55:24.297798 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.288612    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a527b3f0-9f65-4d15-b849-432b1bb3e445-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a527b3f0-9f65-4d15-b849-432b1bb3e445-plugins-config-volume\") pod \"a527b3f0-9f65-4d15-b849-432b1bb3e445\" (UID: \"a527b3f0-9f65-4d15-b849-432b1bb3e445\") "
Nov 03 19:55:24.297798 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.289069    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a527b3f0-9f65-4d15-b849-432b1bb3e445-kube-api-access-7c4sq\" (UniqueName: \"kubernetes.io/projected/a527b3f0-9f65-4d15-b849-432b1bb3e445-kube-api-access-7c4sq\") pod \"a527b3f0-9f65-4d15-b849-432b1bb3e445\" (UID: \"a527b3f0-9f65-4d15-b849-432b1bb3e445\") "
Nov 03 19:55:24.324242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.317490    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a527b3f0-9f65-4d15-b849-432b1bb3e445-plugins-config-volume" pod "a527b3f0-9f65-4d15-b849-432b1bb3e445" (UID: "a527b3f0-9f65-4d15-b849-432b1bb3e445"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:24.418193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.415094    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a527b3f0-9f65-4d15-b849-432b1bb3e445-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:24.481121 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.475362    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a527b3f0-9f65-4d15-b849-432b1bb3e445-kube-api-access-7c4sq" pod "a527b3f0-9f65-4d15-b849-432b1bb3e445" (UID: "a527b3f0-9f65-4d15-b849-432b1bb3e445"). InnerVolumeSpecName "kube-api-access-7c4sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:24.481121 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.476525    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:55:24.518388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.516766    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5555cb-2681-4a70-af23-ba1281bddc74-kube-api-access-5mstg\" (UniqueName: \"kubernetes.io/projected/7a5555cb-2681-4a70-af23-ba1281bddc74-kube-api-access-5mstg\") pod \"7a5555cb-2681-4a70-af23-ba1281bddc74\" (UID: \"7a5555cb-2681-4a70-af23-ba1281bddc74\") "
Nov 03 19:55:24.518388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.516912    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5555cb-2681-4a70-af23-ba1281bddc74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5555cb-2681-4a70-af23-ba1281bddc74-plugins-config-volume\") pod \"7a5555cb-2681-4a70-af23-ba1281bddc74\" (UID: \"7a5555cb-2681-4a70-af23-ba1281bddc74\") "
Nov 03 19:55:24.518388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.517909    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c4sq\" (UniqueName: \"kubernetes.io/projected/a527b3f0-9f65-4d15-b849-432b1bb3e445-kube-api-access-7c4sq\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:24.518388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.518349    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5555cb-2681-4a70-af23-ba1281bddc74-plugins-config-volume" pod "7a5555cb-2681-4a70-af23-ba1281bddc74" (UID: "7a5555cb-2681-4a70-af23-ba1281bddc74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:24.530388 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.530354    4531 generic.go:356] "Generic (PLEG): container finished" podID="a527b3f0-9f65-4d15-b849-432b1bb3e445" containerID="6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251" exitCode=0
Nov 03 19:55:24.530811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.530787    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt" event={"ID":"a527b3f0-9f65-4d15-b849-432b1bb3e445","Type":"ContainerDied","Data":"6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251"}
Nov 03 19:55:24.530964 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.530950    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt" event={"ID":"a527b3f0-9f65-4d15-b849-432b1bb3e445","Type":"ContainerDied","Data":"c2319c7968e7cf7e04d4fc42f6db8f9a70309c70b8884162eafa7b814d347edf"}
Nov 03 19:55:24.531060 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.531048    4531 scope.go:122] "RemoveContainer" containerID="6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251"
Nov 03 19:55:24.531337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.531319    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"
Nov 03 19:55:24.536142 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.536109    4531 generic.go:356] "Generic (PLEG): container finished" podID="7a5555cb-2681-4a70-af23-ba1281bddc74" containerID="3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37" exitCode=0
Nov 03 19:55:24.536279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.536200    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8" event={"ID":"7a5555cb-2681-4a70-af23-ba1281bddc74","Type":"ContainerDied","Data":"3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37"}
Nov 03 19:55:24.536279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.536249    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8" event={"ID":"7a5555cb-2681-4a70-af23-ba1281bddc74","Type":"ContainerDied","Data":"db86b72c75096f541afe9d80d8d55ff9cd77ce04e9220fab4f93cf435119fe86"}
Nov 03 19:55:24.536469 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.536449    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"
Nov 03 19:55:24.553646 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.553457    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5555cb-2681-4a70-af23-ba1281bddc74-kube-api-access-5mstg" pod "7a5555cb-2681-4a70-af23-ba1281bddc74" (UID: "7a5555cb-2681-4a70-af23-ba1281bddc74"). InnerVolumeSpecName "kube-api-access-5mstg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:24.560665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.560645    4531 scope.go:122] "RemoveContainer" containerID="6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251"
Nov 03 19:55:24.561038 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:55:24.561009    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251\": container with ID starting with 6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251 not found: ID does not exist" containerID="6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251"
Nov 03 19:55:24.561112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.561052    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251"} err="failed to get container status \"6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251\": rpc error: code = NotFound desc = could not find container \"6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251\": container with ID starting with 6e87927be2f7dafe55e6aaa89a6bdb285260e8f24efeff26bbabb322497f0251 not found: ID does not exist"
Nov 03 19:55:24.561112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.561078    4531 scope.go:122] "RemoveContainer" containerID="3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37"
Nov 03 19:55:24.571585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.571562    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"]
Nov 03 19:55:24.575530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.574944    4531 scope.go:122] "RemoveContainer" containerID="3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37"
Nov 03 19:55:24.575530 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:55:24.575306    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37\": container with ID starting with 3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37 not found: ID does not exist" containerID="3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37"
Nov 03 19:55:24.575530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.575340    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37"} err="failed to get container status \"3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37\": rpc error: code = NotFound desc = could not find container \"3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37\": container with ID starting with 3f802b602e1fa3fe29f1ee1ef3a14eb4152293dc9d356f0ec34651f45e0fbc37 not found: ID does not exist"
Nov 03 19:55:24.579375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.579352    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c8eec-56cf9b4c9f-pw4gt"]
Nov 03 19:55:24.620263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.620235    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mstg\" (UniqueName: \"kubernetes.io/projected/7a5555cb-2681-4a70-af23-ba1281bddc74-kube-api-access-5mstg\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:24.620263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.620257    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5555cb-2681-4a70-af23-ba1281bddc74-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:24.853256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.848824    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:24.914268 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.914148    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a527b3f0-9f65-4d15-b849-432b1bb3e445" path="/var/lib/kubelet/pods/a527b3f0-9f65-4d15-b849-432b1bb3e445/volumes"
Nov 03 19:55:24.980109 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.980078    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"]
Nov 03 19:55:24.990746 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:24.990718    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c8eec-5d6fcc5bf5-xtjr8"]
Nov 03 19:55:25.024516 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:25.024484    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:25.170598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:25.170568    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:26.948959 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:26.948922    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5555cb-2681-4a70-af23-ba1281bddc74" path="/var/lib/kubelet/pods/7a5555cb-2681-4a70-af23-ba1281bddc74/volumes"
Nov 03 19:55:27.212719 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.212644    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"]
Nov 03 19:55:27.213676 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.213650    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a527b3f0-9f65-4d15-b849-432b1bb3e445" containerName="payload-processing"
Nov 03 19:55:27.213676 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.213671    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a527b3f0-9f65-4d15-b849-432b1bb3e445" containerName="payload-processing"
Nov 03 19:55:27.213854 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.213692    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7a5555cb-2681-4a70-af23-ba1281bddc74" containerName="payload-pre-processing"
Nov 03 19:55:27.213854 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.213697    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5555cb-2681-4a70-af23-ba1281bddc74" containerName="payload-pre-processing"
Nov 03 19:55:27.213928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.213862    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5555cb-2681-4a70-af23-ba1281bddc74" containerName="payload-pre-processing"
Nov 03 19:55:27.213928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.213883    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a527b3f0-9f65-4d15-b849-432b1bb3e445" containerName="payload-processing"
Nov 03 19:55:27.219558 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.219535    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 19:55:27.220826 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.220799    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:27.226521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.226494    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"]
Nov 03 19:55:27.233552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.233525    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-79c47f\""
Nov 03 19:55:27.238097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.238045    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"]
Nov 03 19:55:27.244709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.244671    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 19:55:27.252690 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.252657    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-79c47f-dockercfg-8dw4d\""
Nov 03 19:55:27.270326 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.270294    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"]
Nov 03 19:55:27.289951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.289914    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp4zn\" (UniqueName: \"kubernetes.io/projected/c9d1da86-ac7c-43eb-8381-142219ae2e9b-kube-api-access-pp4zn\") pod \"payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg\" (UID: \"c9d1da86-ac7c-43eb-8381-142219ae2e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 19:55:27.290248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.290224    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d1da86-ac7c-43eb-8381-142219ae2e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg\" (UID: \"c9d1da86-ac7c-43eb-8381-142219ae2e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 19:55:27.290371 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.290269    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98j2\" (UniqueName: \"kubernetes.io/projected/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-kube-api-access-b98j2\") pod \"payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws\" (UID: \"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 19:55:27.290470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.290450    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws\" (UID: \"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 19:55:27.354228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.354145    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:27.392979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.392947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws\" (UID: \"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 19:55:27.393157 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.393114    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp4zn\" (UniqueName: \"kubernetes.io/projected/c9d1da86-ac7c-43eb-8381-142219ae2e9b-kube-api-access-pp4zn\") pod \"payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg\" (UID: \"c9d1da86-ac7c-43eb-8381-142219ae2e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 19:55:27.393289 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.393276    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d1da86-ac7c-43eb-8381-142219ae2e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg\" (UID: \"c9d1da86-ac7c-43eb-8381-142219ae2e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 19:55:27.393332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.393300    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98j2\" (UniqueName: \"kubernetes.io/projected/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-kube-api-access-b98j2\") pod \"payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws\" (UID: \"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 19:55:27.393787 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.393767    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws\" (UID: \"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 19:55:27.394393 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.394373    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d1da86-ac7c-43eb-8381-142219ae2e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg\" (UID: \"c9d1da86-ac7c-43eb-8381-142219ae2e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 19:55:27.404521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.404487    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98j2\" (UniqueName: \"kubernetes.io/projected/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-kube-api-access-b98j2\") pod \"payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws\" (UID: \"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 19:55:27.404636 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.404543    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp4zn\" (UniqueName: \"kubernetes.io/projected/c9d1da86-ac7c-43eb-8381-142219ae2e9b-kube-api-access-pp4zn\") pod \"payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg\" (UID: \"c9d1da86-ac7c-43eb-8381-142219ae2e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 19:55:27.554246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.551919    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 19:55:27.641839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:27.641807    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 19:55:28.054198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.048501    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:28.417640 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:55:28.417599    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958e2cae_1ff0_4e97_8bd9_a9fbdac37dcb.slice/crio-963d1c2bc8dcb58842f9ecf0377b4d36be9df05aa26379804fd70110708507d7 WatchSource:0}: Error finding container 963d1c2bc8dcb58842f9ecf0377b4d36be9df05aa26379804fd70110708507d7: Status 404 returned error can't find the container with id 963d1c2bc8dcb58842f9ecf0377b4d36be9df05aa26379804fd70110708507d7
Nov 03 19:55:28.420149 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.420120    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"]
Nov 03 19:55:28.463768 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.463742    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"]
Nov 03 19:55:28.621981 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.621940    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:28.817870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.817786    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:28.952349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.952315    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws" event={"ID":"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb","Type":"ContainerStarted","Data":"8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633"}
Nov 03 19:55:28.952546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.952373    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws" event={"ID":"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb","Type":"ContainerStarted","Data":"963d1c2bc8dcb58842f9ecf0377b4d36be9df05aa26379804fd70110708507d7"}
Nov 03 19:55:28.952546 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.952410    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 19:55:28.954308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.954281    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg" event={"ID":"c9d1da86-ac7c-43eb-8381-142219ae2e9b","Type":"ContainerStarted","Data":"392997bd1370ed3e58aa0a575a72016e18f86f29703bb39eacb33300352a77e8"}
Nov 03 19:55:28.954464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.954345    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg" event={"ID":"c9d1da86-ac7c-43eb-8381-142219ae2e9b","Type":"ContainerStarted","Data":"6e1b91839ea890572f04e0cff1941201b176ba032121f0d547f5f6492c63b2bf"}
Nov 03 19:55:28.954464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.954374    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 19:55:28.972514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.972465    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws" podStartSLOduration=1.9724517499999998 podStartE2EDuration="1.97245175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:28.968158886 +0000 UTC m=+1442.238282194" watchObservedRunningTime="2026-11-03 19:55:28.97245175 +0000 UTC m=+1442.242575044"
Nov 03 19:55:28.996559 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:28.996502    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg" podStartSLOduration=1.9964845009999999 podStartE2EDuration="1.996484501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:28.992871331 +0000 UTC m=+1442.262994628" watchObservedRunningTime="2026-11-03 19:55:28.996484501 +0000 UTC m=+1442.266607799"
Nov 03 19:55:29.887187 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:29.887141    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:29.916118 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:29.916082    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"]
Nov 03 19:55:29.921709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:29.921688    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 19:55:29.924701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:29.924675    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7a62d4-dockercfg-zn47f\""
Nov 03 19:55:29.927314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:29.927294    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7a62d4\""
Nov 03 19:55:29.929873 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:29.929850    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"]
Nov 03 19:55:29.951597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:29.951561    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"]
Nov 03 19:55:29.962456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:29.961649    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"]
Nov 03 19:55:29.962456 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:29.961895    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 19:55:30.102203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.042009    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:30.116198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.115802    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb\" (UID: \"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 19:55:30.116198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.115858    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vfr\" (UniqueName: \"kubernetes.io/projected/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-kube-api-access-27vfr\") pod \"payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb\" (UID: \"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 19:55:30.274247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.221678    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb\" (UID: \"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 19:55:30.274247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.221735    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27vfr\" (UniqueName: \"kubernetes.io/projected/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-kube-api-access-27vfr\") pod \"payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb\" (UID: \"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 19:55:30.274247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.221922    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwzw\" (UniqueName: \"kubernetes.io/projected/762a9a66-e013-496b-b744-dc20fe6d4b13-kube-api-access-hrwzw\") pod \"payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4\" (UID: \"762a9a66-e013-496b-b744-dc20fe6d4b13\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 19:55:30.274247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.222101    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/762a9a66-e013-496b-b744-dc20fe6d4b13-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4\" (UID: \"762a9a66-e013-496b-b744-dc20fe6d4b13\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 19:55:30.274247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.223606    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb\" (UID: \"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 19:55:30.357199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.340639    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/762a9a66-e013-496b-b744-dc20fe6d4b13-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4\" (UID: \"762a9a66-e013-496b-b744-dc20fe6d4b13\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 19:55:30.357199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.347248    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vfr\" (UniqueName: \"kubernetes.io/projected/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-kube-api-access-27vfr\") pod \"payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb\" (UID: \"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 19:55:30.361197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.359001    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwzw\" (UniqueName: \"kubernetes.io/projected/762a9a66-e013-496b-b744-dc20fe6d4b13-kube-api-access-hrwzw\") pod \"payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4\" (UID: \"762a9a66-e013-496b-b744-dc20fe6d4b13\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 19:55:30.361197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.360354    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/762a9a66-e013-496b-b744-dc20fe6d4b13-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4\" (UID: \"762a9a66-e013-496b-b744-dc20fe6d4b13\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 19:55:30.372974 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.369278    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwzw\" (UniqueName: \"kubernetes.io/projected/762a9a66-e013-496b-b744-dc20fe6d4b13-kube-api-access-hrwzw\") pod \"payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4\" (UID: \"762a9a66-e013-496b-b744-dc20fe6d4b13\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 19:55:30.522274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.512080    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:30.561759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.545202    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 19:55:30.619822 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.619780    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 19:55:30.931815 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:55:30.931787    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41a7e6a_0f0c_4d38_ac4a_bb5b534cdda3.slice/crio-02e5d8ae08128a43b419694435a2233d775bcca719cc62595376159c372eb0b7 WatchSource:0}: Error finding container 02e5d8ae08128a43b419694435a2233d775bcca719cc62595376159c372eb0b7: Status 404 returned error can't find the container with id 02e5d8ae08128a43b419694435a2233d775bcca719cc62595376159c372eb0b7
Nov 03 19:55:30.934088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.934061    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"]
Nov 03 19:55:30.959551 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:55:30.959522    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762a9a66_e013_496b_b744_dc20fe6d4b13.slice/crio-5fdf84f0013cbf73c6c7ac7c5078e7488c59de3ee494b78ae76b26eb5f8fafc5 WatchSource:0}: Error finding container 5fdf84f0013cbf73c6c7ac7c5078e7488c59de3ee494b78ae76b26eb5f8fafc5: Status 404 returned error can't find the container with id 5fdf84f0013cbf73c6c7ac7c5078e7488c59de3ee494b78ae76b26eb5f8fafc5
Nov 03 19:55:30.960357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.960327    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"]
Nov 03 19:55:30.984088 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.984047    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4" event={"ID":"762a9a66-e013-496b-b744-dc20fe6d4b13","Type":"ContainerStarted","Data":"5fdf84f0013cbf73c6c7ac7c5078e7488c59de3ee494b78ae76b26eb5f8fafc5"}
Nov 03 19:55:30.985664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:30.985638    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb" event={"ID":"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3","Type":"ContainerStarted","Data":"02e5d8ae08128a43b419694435a2233d775bcca719cc62595376159c372eb0b7"}
Nov 03 19:55:31.997032 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:31.996996    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb" event={"ID":"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3","Type":"ContainerStarted","Data":"ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b"}
Nov 03 19:55:31.997512 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:31.997076    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 19:55:31.999740 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:31.999710    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4" event={"ID":"762a9a66-e013-496b-b744-dc20fe6d4b13","Type":"ContainerStarted","Data":"43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a"}
Nov 03 19:55:31.999880 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:31.999766    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 19:55:32.033336 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.033288    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb" podStartSLOduration=3.033272562 podStartE2EDuration="3.033272562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:32.028045532 +0000 UTC m=+1445.298168826" watchObservedRunningTime="2026-11-03 19:55:32.033272562 +0000 UTC m=+1445.303395857"
Nov 03 19:55:32.050817 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.050755    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4" podStartSLOduration=3.050737117 podStartE2EDuration="3.050737117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:32.043798555 +0000 UTC m=+1445.313921863" watchObservedRunningTime="2026-11-03 19:55:32.050737117 +0000 UTC m=+1445.320860412"
Nov 03 19:55:32.303131 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.303015    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:32.782848 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.782811    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"]
Nov 03 19:55:32.788635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.788594    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:32.795898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.795863    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"]
Nov 03 19:55:32.922389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.922355    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:32.922591 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.922396    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:32.922674 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.922602    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:32.922674 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.922643    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:32.922674 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.922667    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46k5r\" (UniqueName: \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-kube-api-access-46k5r\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:32.922849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.922792    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:32.922924 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.922907    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:32.923529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.923324    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:32.923529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:32.923453    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.025944 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.025907    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026106    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026136    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026166    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026225    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026249    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46k5r\" (UniqueName: \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-kube-api-access-46k5r\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026288    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026342    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026429    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026535    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026592    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026737    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.026951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.026786    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.027444 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.027421    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.028846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.028825    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.028989 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.028970    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.032987 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.032926    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.033378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.033357    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46k5r\" (UniqueName: \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-kube-api-access-46k5r\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-hs829\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.108875 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.108832    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:33.400654 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:55:33.400621    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7ebaff_5e88_4e7e_a0d0_e446144c29f9.slice/crio-7e30ae2778a9b4e4848e11a425db299e960d10cc0516012e99aff6b9f8c353f0 WatchSource:0}: Error finding container 7e30ae2778a9b4e4848e11a425db299e960d10cc0516012e99aff6b9f8c353f0: Status 404 returned error can't find the container with id 7e30ae2778a9b4e4848e11a425db299e960d10cc0516012e99aff6b9f8c353f0
Nov 03 19:55:33.403332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.403303    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"]
Nov 03 19:55:33.403977 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.403940    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:33.404086 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.404029    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:33.404086 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.404076    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:33.690735 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.690659    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"]
Nov 03 19:55:33.696949 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.696758    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:33.707197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.700052    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:55:33.707197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.700122    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:55:33.707197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.701115    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"]
Nov 03 19:55:33.744113 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.744072    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cqr4\" (UniqueName: \"kubernetes.io/projected/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-kube-api-access-8cqr4\") pod \"maas-api-revoke-sub-d5819b09-4rtsm\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:33.744679 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.744375    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5819b09-4rtsm\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:33.744679 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.744533    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5819b09-4rtsm\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:33.847036 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.847003    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5819b09-4rtsm\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:33.847184 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.847108    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5819b09-4rtsm\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:33.847359 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.847339    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cqr4\" (UniqueName: \"kubernetes.io/projected/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-kube-api-access-8cqr4\") pod \"maas-api-revoke-sub-d5819b09-4rtsm\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:33.847763 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.847741    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5819b09-4rtsm\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:33.850438 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.850417    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5819b09-4rtsm\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:33.855161 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:33.855141    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cqr4\" (UniqueName: \"kubernetes.io/projected/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-kube-api-access-8cqr4\") pod \"maas-api-revoke-sub-d5819b09-4rtsm\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:34.041370 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:34.037126    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:34.047430 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:34.047373    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" event={"ID":"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9","Type":"ContainerStarted","Data":"73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434"}
Nov 03 19:55:34.047577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:34.047440    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" event={"ID":"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9","Type":"ContainerStarted","Data":"7e30ae2778a9b4e4848e11a425db299e960d10cc0516012e99aff6b9f8c353f0"}
Nov 03 19:55:34.071187 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:34.071122    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" podStartSLOduration=2.0711028479999998 podStartE2EDuration="2.071102848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:34.066781672 +0000 UTC m=+1447.336904968" watchObservedRunningTime="2026-11-03 19:55:34.071102848 +0000 UTC m=+1447.341226144"
Nov 03 19:55:34.109223 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:34.108978    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:34.111617 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:34.111586    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-hs829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 19:55:34.111820 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:34.111637    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 19:55:34.339918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:34.339768    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:34.400568 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:55:34.400526    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0e8797_c7b1_4e3c_aafa_ffdb6d5f920c.slice/crio-f3b3a5d69260ac6ee78e71d1230f635c1ba369fc011b1b01305acb311be35da6 WatchSource:0}: Error finding container f3b3a5d69260ac6ee78e71d1230f635c1ba369fc011b1b01305acb311be35da6: Status 404 returned error can't find the container with id f3b3a5d69260ac6ee78e71d1230f635c1ba369fc011b1b01305acb311be35da6
Nov 03 19:55:34.402310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:34.401124    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"]
Nov 03 19:55:35.076710 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:35.076677    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm" event={"ID":"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c","Type":"ContainerStarted","Data":"f3b3a5d69260ac6ee78e71d1230f635c1ba369fc011b1b01305acb311be35da6"}
Nov 03 19:55:35.110303 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:35.110275    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-hs829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 19:55:35.110442 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:35.110322    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 19:55:35.216209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:35.215958    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:36.087537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:36.087446    4531 generic.go:356] "Generic (PLEG): container finished" podID="fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c" containerID="a763bcde7924dadde28ab95c476bcf85944d27ca28292c6b59047268b2ea66f7" exitCode=0
Nov 03 19:55:36.087912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:36.087525    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm" event={"ID":"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c","Type":"ContainerDied","Data":"a763bcde7924dadde28ab95c476bcf85944d27ca28292c6b59047268b2ea66f7"}
Nov 03 19:55:36.120198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:36.113304    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-hs829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 19:55:36.120198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:36.113361    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 19:55:36.123893 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:36.123239    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:36.977189 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:36.977102    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:37.124194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.122550    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-hs829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 19:55:37.124194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.122607    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 19:55:37.389023 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.388995    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:37.484459 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.484359    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-openshift-ca\") pod \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") "
Nov 03 19:55:37.484651 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.484534    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-kube-api-access-8cqr4\" (UniqueName: \"kubernetes.io/projected/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-kube-api-access-8cqr4\") pod \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") "
Nov 03 19:55:37.484732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.484651    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-xks-ca\") pod \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\" (UID: \"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c\") "
Nov 03 19:55:37.484811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.484785    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-openshift-ca" pod "fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c" (UID: "fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:37.485535 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.485511    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:37.486898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.486875    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-kube-api-access-8cqr4" pod "fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c" (UID: "fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c"). InnerVolumeSpecName "kube-api-access-8cqr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:37.487323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.487294    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-xks-ca" pod "fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c" (UID: "fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:55:37.587980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.587946    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:37.587980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.587978    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cqr4\" (UniqueName: \"kubernetes.io/projected/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c-kube-api-access-8cqr4\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:37.722119 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:37.722078    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:38.109455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:38.109421    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-hs829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 19:55:38.109651 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:38.109503    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 19:55:38.170716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:38.170675    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm" event={"ID":"fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c","Type":"ContainerDied","Data":"f3b3a5d69260ac6ee78e71d1230f635c1ba369fc011b1b01305acb311be35da6"}
Nov 03 19:55:38.171297 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:38.170753    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b3a5d69260ac6ee78e71d1230f635c1ba369fc011b1b01305acb311be35da6"
Nov 03 19:55:38.171297 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:38.170824    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"
Nov 03 19:55:38.563522 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:38.563483    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:39.109243 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:39.109159    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-hs829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 19:55:39.109414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:39.109271    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 19:55:39.512668 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:39.508282    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:39.976713 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:39.976680    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 19:55:39.976894 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:39.976779    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 19:55:40.109318 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.109280    4531 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-hs829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 19:55:40.109531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.109345    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 19:55:40.171678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.171643    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"]
Nov 03 19:55:40.172957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.172927    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c" containerName="revoke-keys"
Nov 03 19:55:40.172957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.172954    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c" containerName="revoke-keys"
Nov 03 19:55:40.173393 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.173374    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c" containerName="revoke-keys"
Nov 03 19:55:40.177976 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.177950    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.181355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.181333    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-225668-openshift-default-dockercfg-xmp27\""
Nov 03 19:55:40.188415 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.188391    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"]
Nov 03 19:55:40.301574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.301470    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdfkg\" (UniqueName: \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-kube-api-access-wdfkg\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.301574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.301530    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-certs\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.301765 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.301676    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74560e30-953f-48f5-8031-9a73b7218cef-istio-podinfo\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.301811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.301781    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-credential-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.302019 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.301991    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-envoy\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.302229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.302156    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-data\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.302301 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.302233    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.302373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.302348    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74560e30-953f-48f5-8031-9a73b7218cef-istiod-ca-cert\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.302464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.302449    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-istio-token\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.403858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.403822    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-data\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.404043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.403930    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.404043 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.403984    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74560e30-953f-48f5-8031-9a73b7218cef-istiod-ca-cert\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.404113 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.404046    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-istio-token\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.404193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.404163    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdfkg\" (UniqueName: \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-kube-api-access-wdfkg\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.404269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.404216    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-certs\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.404269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.404247    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74560e30-953f-48f5-8031-9a73b7218cef-istio-podinfo\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.404378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.404310    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-credential-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.404378 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.404349    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-data\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.404506 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.404427    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.404566 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.404547    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-envoy\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.404860 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.404837    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74560e30-953f-48f5-8031-9a73b7218cef-istiod-ca-cert\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.405020 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.404997    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-certs\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.405020 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.405014    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-credential-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.406965 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.406943    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74560e30-953f-48f5-8031-9a73b7218cef-istio-podinfo\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.407055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.406964    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-envoy\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.412466 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.412443    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-istio-token\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.412797 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.412776    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdfkg\" (UniqueName: \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-kube-api-access-wdfkg\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-2mjrh\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.454314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.449629    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:40.517623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.517586    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:40.860649 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:55:40.860612    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74560e30_953f_48f5_8031_9a73b7218cef.slice/crio-f3b82e8eaf6a47966545804fd9db6e4170b39cabc8d4ba7caf444b7f6ef3a389 WatchSource:0}: Error finding container f3b82e8eaf6a47966545804fd9db6e4170b39cabc8d4ba7caf444b7f6ef3a389: Status 404 returned error can't find the container with id f3b82e8eaf6a47966545804fd9db6e4170b39cabc8d4ba7caf444b7f6ef3a389
Nov 03 19:55:40.864428 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.864387    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:40.864537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.864477    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:40.864537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.864520    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:40.864626 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:40.864596    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"]
Nov 03 19:55:41.115472 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:41.115395    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:41.115913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:41.115897    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:41.116734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:41.116712    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 19:55:41.235358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:41.235319    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" event={"ID":"74560e30-953f-48f5-8031-9a73b7218cef","Type":"ContainerStarted","Data":"75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37"}
Nov 03 19:55:41.235635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:41.235609    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" event={"ID":"74560e30-953f-48f5-8031-9a73b7218cef","Type":"ContainerStarted","Data":"f3b82e8eaf6a47966545804fd9db6e4170b39cabc8d4ba7caf444b7f6ef3a389"}
Nov 03 19:55:41.243486 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:41.243133    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:41.258385 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:41.258328    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" podStartSLOduration=1.258312178 podStartE2EDuration="1.258312178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:41.25224187 +0000 UTC m=+1454.522365166" watchObservedRunningTime="2026-11-03 19:55:41.258312178 +0000 UTC m=+1454.528435473"
Nov 03 19:55:41.517778 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:41.517746    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:41.518957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:41.518930    4531 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 19:55:41.519101 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:41.518980    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" podUID="74560e30-953f-48f5-8031-9a73b7218cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 19:55:42.135271 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:42.122602    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:42.518245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:42.518118    4531 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 19:55:42.518245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:42.518209    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" podUID="74560e30-953f-48f5-8031-9a73b7218cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 19:55:42.897429 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:42.897368    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:43.070261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:43.068433    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 19:55:43.070261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:43.068620    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 19:55:43.518499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:43.518458    4531 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 19:55:43.518891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:43.518517    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" podUID="74560e30-953f-48f5-8031-9a73b7218cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 19:55:43.602489 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:43.602443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:44.267197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:44.252070    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:44.390845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:44.390805    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:44.518477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:44.518387    4531 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 19:55:44.518477 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:44.518443    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" podUID="74560e30-953f-48f5-8031-9a73b7218cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 19:55:44.856470 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:44.855088    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:45.396746 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:45.396710    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:45.536197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:45.535352    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:45.555273 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:45.555215    4531 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 19:55:45.555454 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:45.555346    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" podUID="74560e30-953f-48f5-8031-9a73b7218cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 19:55:46.519779 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:46.519740    4531 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 19:55:46.519971 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:46.519827    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" podUID="74560e30-953f-48f5-8031-9a73b7218cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 19:55:46.933714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:46.933671    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:47.099922 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.099887    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"]
Nov 03 19:55:47.106187 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.106144    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.111515 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.111489    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bdb694-openshift-default-dockercfg-jxjgx\""
Nov 03 19:55:47.119154 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.119117    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"]
Nov 03 19:55:47.220616 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.220510    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-credential-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.220772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.220657    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-podinfo\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.220772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.220686    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77fwr\" (UniqueName: \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-kube-api-access-77fwr\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.220772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.220738    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-certs\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.220772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.220768    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.220922 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.220829    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-envoy\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.220922 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.220853    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-token\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.220992 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.220950    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-data\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.220992 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.220974    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istiod-ca-cert\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.322777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.322737    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-certs\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.322934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.322789    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.322934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.322852    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-envoy\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.323057 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.322980    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-token\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.323115 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.323078    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-data\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.323115 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.323102    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istiod-ca-cert\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.323281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.323112    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-certs\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.323281 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.323238    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.323389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.323347    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-credential-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.323453 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.323428    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-data\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.323510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.323466    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-podinfo\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.323510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.323500    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77fwr\" (UniqueName: \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-kube-api-access-77fwr\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.323683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.323664    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-credential-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.323846 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.323824    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istiod-ca-cert\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.325362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.325340    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-envoy\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.325801 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.325783    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-podinfo\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.330867 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.330845    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-token\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.331233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.331184    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77fwr\" (UniqueName: \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-kube-api-access-77fwr\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.430380 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.430343    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:47.518047 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.518008    4531 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 19:55:47.518192 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.518070    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" podUID="74560e30-953f-48f5-8031-9a73b7218cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 19:55:47.770325 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.770284    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"]
Nov 03 19:55:47.771409 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:55:47.771374    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a53a089_f3db_4d21_ab7b_f09d17aa4fc8.slice/crio-603efe6f9a62fbc33fc6bac814a5dccc7c86f3eeb877968055726c7b98b364c1 WatchSource:0}: Error finding container 603efe6f9a62fbc33fc6bac814a5dccc7c86f3eeb877968055726c7b98b364c1: Status 404 returned error can't find the container with id 603efe6f9a62fbc33fc6bac814a5dccc7c86f3eeb877968055726c7b98b364c1
Nov 03 19:55:47.775221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.775161    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:47.775353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.775269    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:47.775353 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:47.775314    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:48.433148 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:48.433107    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" event={"ID":"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8","Type":"ContainerStarted","Data":"a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab"}
Nov 03 19:55:48.433577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:48.433168    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" event={"ID":"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8","Type":"ContainerStarted","Data":"603efe6f9a62fbc33fc6bac814a5dccc7c86f3eeb877968055726c7b98b364c1"}
Nov 03 19:55:48.468397 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:48.468316    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" podStartSLOduration=1.4682960299999999 podStartE2EDuration="1.46829603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:48.452419688 +0000 UTC m=+1461.722542999" watchObservedRunningTime="2026-11-03 19:55:48.46829603 +0000 UTC m=+1461.738419331"
Nov 03 19:55:48.524295 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:48.524262    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:48.524748 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:48.524733    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:48.525447 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:48.525424    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 19:55:48.891095 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:48.889291    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:49.074220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:49.073057    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:49.203638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:49.203551    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:49.430983 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:49.430949    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:49.432729 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:49.432695    4531 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 19:55:49.432898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:49.432757    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 19:55:50.431152 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:50.431116    4531 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 19:55:50.431576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:50.431220    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 19:55:50.607219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:50.604984    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:50.613522 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:55:50.613472    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:55:50.904355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:50.852837    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:51.266858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:51.248698    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:51.432137 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:51.432100    4531 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 19:55:51.432586 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:51.432182    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 19:55:51.700893 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:51.700796    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:52.163305 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:52.159741    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:52.293499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:52.293466    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:52.431247 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:52.431124    4531 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 19:55:52.431422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:52.431244    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 19:55:52.449213 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:52.438697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:53.431277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:53.431242    4531 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 19:55:53.431462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:53.431306    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 19:55:54.207339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:54.207301    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:54.431811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:54.431772    4531 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Nov 03 19:55:54.431985 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:54.431848    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Nov 03 19:55:55.338678 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:55.338638    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:55.457806 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:55.457776    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:55.458328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:55.458301    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:55.459112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:55.459092    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 19:55:56.242425 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:56.242392    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:56.379241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:56.379205    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:57.419561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:57.418043    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:58.519962 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.519924    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:58.555375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.555337    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"]
Nov 03 19:55:58.579700 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.579670    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"]
Nov 03 19:55:58.579867 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.579785    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.585584 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.585562    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7a62d4-serving-cert\""
Nov 03 19:55:58.666274 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.666102    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.666837 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.666793    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcc8j\" (UniqueName: \"kubernetes.io/projected/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-kube-api-access-gcc8j\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.667107 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.667081    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-metrics-certs\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.667203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.667130    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.695964 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.695932    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:58.769237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.769152    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-metrics-certs\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.769553 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.769258    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.769553 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.769422    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.769697 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.769621    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcc8j\" (UniqueName: \"kubernetes.io/projected/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-kube-api-access-gcc8j\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.770231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.770118    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.772751 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.772697    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-metrics-certs\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.774342 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.774316    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-maas-api-tls\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.807196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.806089    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcc8j\" (UniqueName: \"kubernetes.io/projected/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-kube-api-access-gcc8j\") pod \"maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.903203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.901082    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:55:58.982198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:58.975367    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:59.487742 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:55:59.487701    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2692cd61_b0a3_4a74_9f4e_3f4c1ca4e35a.slice/crio-dff5c0a62c522f935e8d4464b06d6e9a73e2e33f68a1d35b72b9482a20d887fd WatchSource:0}: Error finding container dff5c0a62c522f935e8d4464b06d6e9a73e2e33f68a1d35b72b9482a20d887fd: Status 404 returned error can't find the container with id dff5c0a62c522f935e8d4464b06d6e9a73e2e33f68a1d35b72b9482a20d887fd
Nov 03 19:55:59.488652 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:59.488519    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"]
Nov 03 19:55:59.863877 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:59.863787    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d" event={"ID":"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a","Type":"ContainerStarted","Data":"dff5c0a62c522f935e8d4464b06d6e9a73e2e33f68a1d35b72b9482a20d887fd"}
Nov 03 19:55:59.864488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:55:59.864018    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:00.007847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:00.007804    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:00.697186 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:00.697139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:01.034282 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:01.034092    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d" event={"ID":"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a","Type":"ContainerStarted","Data":"896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3"}
Nov 03 19:56:01.034961 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:01.034939    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:56:01.677521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:01.677482    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:02.366140 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:02.366104    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:03.011144 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:03.011112    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:03.731876 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:03.730447    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:04.605103 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:04.605022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:05.330464 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:05.330410    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:06.048196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:06.045836    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:06.760989 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:06.760945    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:07.051902 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:07.051832    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 19:56:07.071369 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:07.071313    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d" podStartSLOduration=8.458083609 podStartE2EDuration="9.071298325s" totalImagesPullingTime="613.214716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:01.173477521 +0000 UTC m=+1474.443600828" watchObservedRunningTime="2026-11-03 19:56:07.071298325 +0000 UTC m=+1480.341421620"
Nov 03 19:56:07.465197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:07.456167    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:08.104120 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:08.104071    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:08.719481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:08.719445    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:09.461901 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:09.460961    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:10.233414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:10.233381    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:10.891200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:10.888025    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:11.538499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:11.538456    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:12.134008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:12.133974    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:12.293986 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:12.293949    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:12.565919 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:12.565788    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:12.750529 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:12.750490    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"]
Nov 03 19:56:12.755291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:12.755260    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c8eec-t699p"]
Nov 03 19:56:12.863516 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:12.863437    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7758898d-f2e3-4d18-af29-34326ffae000" path="/var/lib/kubelet/pods/7758898d-f2e3-4d18-af29-34326ffae000/volumes"
Nov 03 19:56:12.889262 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:12.889227    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:13.270147 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:13.270106    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:13.831199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:13.831003    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:15.305667 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:15.305631    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:15.504337 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:15.504301    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:16.165623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:16.165587    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:16.424253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:16.423872    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:17.036310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:17.036272    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:17.875835 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:17.875794    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:18.531566 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:18.531534    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:19.445710 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:19.445671    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:19.601727 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:19.601689    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:19.694633 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:19.694598    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"]
Nov 03 19:56:19.695104 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:19.695072    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy" containerID="cri-o://198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e" gracePeriod=30
Nov 03 19:56:19.763264 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:19.763163    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:20.053251 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:20.049306    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:20.578708 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:56:20.578667    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:56:20.967808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:20.967779    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:21.655283 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:21.655244    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:22.163665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.163628    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"]
Nov 03 19:56:22.172570 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.172539    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.175566 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.175538    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-225668\"/\"openshift-service-ca.crt\""
Nov 03 19:56:22.175745 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.175729    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-225668\"/\"kube-root-ca.crt\""
Nov 03 19:56:22.175868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.175854    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-225668\"/\"default-dockercfg-gpz56\""
Nov 03 19:56:22.176130 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.176107    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-225668\"/\"test-model-225668-kserve-self-signed-certs\""
Nov 03 19:56:22.176789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.176766    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"]
Nov 03 19:56:22.229444 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.229414    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:22.351750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.351705    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pd4r\" (UniqueName: \"kubernetes.io/projected/4a54f455-59f9-410c-8b6a-8c56f0828800-kube-api-access-9pd4r\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.352154 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.352127    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a54f455-59f9-410c-8b6a-8c56f0828800-tls-certs\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.352519 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.352500    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-model-cache\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.352648 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.352631    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-tmp-dir\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.352916 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.352887    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-dshm\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.353089 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.353071    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-home\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.457255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.457091    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-model-cache\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.457504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.457484    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-tmp-dir\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.457747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.457731    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-dshm\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.458008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.457985    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-home\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.458429 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.458401    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pd4r\" (UniqueName: \"kubernetes.io/projected/4a54f455-59f9-410c-8b6a-8c56f0828800-kube-api-access-9pd4r\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.459123 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.459053    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a54f455-59f9-410c-8b6a-8c56f0828800-tls-certs\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.471189 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.466392    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-home\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.471189 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.466961    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-model-cache\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.471189 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.467229    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-tmp-dir\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.481585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.472579    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a54f455-59f9-410c-8b6a-8c56f0828800-tls-certs\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.504068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.504025    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pd4r\" (UniqueName: \"kubernetes.io/projected/4a54f455-59f9-410c-8b6a-8c56f0828800-kube-api-access-9pd4r\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.528537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.528497    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-dshm\") pod \"test-model-225668-kserve-6fcc8c56df-ltgld\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") " pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:22.790923 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:22.790837    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:23.037129 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:23.037090    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:23.080998 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:23.080971    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"]
Nov 03 19:56:23.081118 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:56:23.080969    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a54f455_59f9_410c_8b6a_8c56f0828800.slice/crio-d9a21e4a9b15ff5ccb065d55d088369f51bbec3013c2af30dd45d2aa3c1dc05c WatchSource:0}: Error finding container d9a21e4a9b15ff5ccb065d55d088369f51bbec3013c2af30dd45d2aa3c1dc05c: Status 404 returned error can't find the container with id d9a21e4a9b15ff5ccb065d55d088369f51bbec3013c2af30dd45d2aa3c1dc05c
Nov 03 19:56:23.708200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:23.707849    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld" event={"ID":"4a54f455-59f9-410c-8b6a-8c56f0828800","Type":"ContainerStarted","Data":"323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087"}
Nov 03 19:56:23.708200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:23.707911    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld" event={"ID":"4a54f455-59f9-410c-8b6a-8c56f0828800","Type":"ContainerStarted","Data":"d9a21e4a9b15ff5ccb065d55d088369f51bbec3013c2af30dd45d2aa3c1dc05c"}
Nov 03 19:56:23.783195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:23.780163    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld" podStartSLOduration=1.780140483 podStartE2EDuration="1.780140483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:23.747604303 +0000 UTC m=+1497.017727599" watchObservedRunningTime="2026-11-03 19:56:23.780140483 +0000 UTC m=+1497.050263779"
Nov 03 19:56:23.818199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:23.814623    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:24.260146 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:24.260110    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:25.101897 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.101870    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:56:25.139734 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.139700    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"]
Nov 03 19:56:25.141845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.141816    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy"
Nov 03 19:56:25.141845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.141843    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy"
Nov 03 19:56:25.142078 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.141860    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy"
Nov 03 19:56:25.142078 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.141867    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy"
Nov 03 19:56:25.142221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.142155    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy"
Nov 03 19:56:25.142221 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.142190    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerName="istio-proxy"
Nov 03 19:56:25.147375 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.147350    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.150845 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.150821    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f58171-serving-cert\""
Nov 03 19:56:25.158108 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.158081    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"]
Nov 03 19:56:25.236543 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.236507    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-token\" (UniqueName: \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-token\") pod \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") "
Nov 03 19:56:25.237269 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.237241    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-podinfo\") pod \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") "
Nov 03 19:56:25.237446 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.237380    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-envoy\") pod \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") "
Nov 03 19:56:25.237533 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.237463    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-kube-api-access-5pdjm\" (UniqueName: \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-kube-api-access-5pdjm\") pod \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") "
Nov 03 19:56:25.237603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.237532    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-data\") pod \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") "
Nov 03 19:56:25.237603 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.237576    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-certs\") pod \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") "
Nov 03 19:56:25.237730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.237684    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-socket\") pod \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") "
Nov 03 19:56:25.237794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.237732    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-credential-socket\") pod \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") "
Nov 03 19:56:25.237915 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.237898    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istiod-ca-cert\") pod \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\" (UID: \"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754\") "
Nov 03 19:56:25.238507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.238295    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-certs" pod "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" (UID: "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:56:25.238507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.238342    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/062411ad-e3e7-4e1d-b7b4-c5dba546e973-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.238507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.238413    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-metrics-certs\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.238743 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.238631    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-maas-api-tls\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.238743 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.238695    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-token" pod "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" (UID: "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:25.239062 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.238863    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wsqz\" (UniqueName: \"kubernetes.io/projected/062411ad-e3e7-4e1d-b7b4-c5dba546e973-kube-api-access-4wsqz\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.239062 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.238910    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-data" pod "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" (UID: "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:56:25.239062 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.238937    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istiod-ca-cert" pod "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" (UID: "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:25.239307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.239117    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-credential-socket" pod "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" (UID: "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:56:25.239424 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.239344    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:25.239424 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.239366    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:25.239424 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.239382    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:25.239424 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.239399    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:25.239424 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.239414    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:25.239865 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.239810    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-socket" pod "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" (UID: "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:56:25.240045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.240020    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-envoy" pod "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" (UID: "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:56:25.240045 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.240037    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-podinfo" pod "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" (UID: "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:56:25.241293 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.241272    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-kube-api-access-5pdjm" pod "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" (UID: "ed9ec0ed-4670-47ca-aa89-dd22c8bd9754"). InnerVolumeSpecName "kube-api-access-5pdjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:25.341402 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.341298    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-maas-api-tls\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.341591 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.341448    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wsqz\" (UniqueName: \"kubernetes.io/projected/062411ad-e3e7-4e1d-b7b4-c5dba546e973-kube-api-access-4wsqz\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.341691 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.341672    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/062411ad-e3e7-4e1d-b7b4-c5dba546e973-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.341756 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.341726    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-metrics-certs\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.341868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.341838    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:25.341868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.341854    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:25.341868 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.341864    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pdjm\" (UniqueName: \"kubernetes.io/projected/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-kube-api-access-5pdjm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:25.341986 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.341879    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:25.342414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.342390    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/062411ad-e3e7-4e1d-b7b4-c5dba546e973-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.344090 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.344066    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-metrics-certs\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.344222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.344136    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-maas-api-tls\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.348313 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.348293    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wsqz\" (UniqueName: \"kubernetes.io/projected/062411ad-e3e7-4e1d-b7b4-c5dba546e973-kube-api-access-4wsqz\") pod \"maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.424733 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.424339    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:25.471197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.470546    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:25.929199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:25.910655    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"]
Nov 03 19:56:26.046195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.023751    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv" event={"ID":"062411ad-e3e7-4e1d-b7b4-c5dba546e973","Type":"ContainerStarted","Data":"3ca2f14cf08263d63ddc4e35d93e6526f2cf29a210c55318adca6409519d6c38"}
Nov 03 19:56:26.046195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.029997    4531 generic.go:356] "Generic (PLEG): container finished" podID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" containerID="198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e" exitCode=0
Nov 03 19:56:26.046195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.037030    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"
Nov 03 19:56:26.053912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.053684    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" event={"ID":"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754","Type":"ContainerDied","Data":"198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e"}
Nov 03 19:56:26.053912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.053763    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4" event={"ID":"ed9ec0ed-4670-47ca-aa89-dd22c8bd9754","Type":"ContainerDied","Data":"1ea17365ee7c30b1dded3afd2b501ee11111c956e53c9d58a12c6ff25ee0b67b"}
Nov 03 19:56:26.053912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.053793    4531 scope.go:122] "RemoveContainer" containerID="198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e"
Nov 03 19:56:26.134389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.134368    4531 scope.go:122] "RemoveContainer" containerID="282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5"
Nov 03 19:56:26.152015 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.151973    4531 scope.go:122] "RemoveContainer" containerID="198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e"
Nov 03 19:56:26.152539 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:56:26.152510    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e\": container with ID starting with 198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e not found: ID does not exist" containerID="198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e"
Nov 03 19:56:26.152677 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.152649    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e"} err="failed to get container status \"198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e\": rpc error: code = NotFound desc = could not find container \"198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e\": container with ID starting with 198d7087edd8a8038aa1c0ebdf7f2bea2e1dfaeda53e0f02af205addf093a48e not found: ID does not exist"
Nov 03 19:56:26.152768 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.152704    4531 scope.go:122] "RemoveContainer" containerID="282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5"
Nov 03 19:56:26.153072 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:56:26.153052    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5\": container with ID starting with 282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5 not found: ID does not exist" containerID="282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5"
Nov 03 19:56:26.153144 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.153081    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5"} err="failed to get container status \"282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5\": rpc error: code = NotFound desc = could not find container \"282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5\": container with ID starting with 282dc2b1bec6a69db5eb4815b5f780146a16782ed3bf7c67aa401496ffe113d5 not found: ID does not exist"
Nov 03 19:56:26.155814 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.155755    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"]
Nov 03 19:56:26.161218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:26.161198    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0c8eec-openshift-default-7ccb89bb4f-tfcl4"]
Nov 03 19:56:27.156207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:27.146948    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9ec0ed-4670-47ca-aa89-dd22c8bd9754" path="/var/lib/kubelet/pods/ed9ec0ed-4670-47ca-aa89-dd22c8bd9754/volumes"
Nov 03 19:56:27.212445 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:27.212410    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:27.259260 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:27.256379    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:27.383194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:27.350098    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv" podStartSLOduration=7.727887547 podStartE2EDuration="8.35008011s" totalImagesPullingTime="622.192563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:27.347369814 +0000 UTC m=+1500.617493107" watchObservedRunningTime="2026-11-03 19:56:27.35008011 +0000 UTC m=+1500.620203405"
Nov 03 19:56:28.319199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:28.315357    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:28.398200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:28.396187    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv" event={"ID":"062411ad-e3e7-4e1d-b7b4-c5dba546e973","Type":"ContainerStarted","Data":"5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b"}
Nov 03 19:56:28.763209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:28.761258    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:28.777026 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:28.776987    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:28.799316 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:28.799265    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:28.864198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:28.863454    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:28.950365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:28.950294    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" podUID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.194:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:28.998742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:28.998693    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:29.063635 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.062779    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:56:29.065839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.064936    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:29.130199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.110706    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:29.131517 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.130493    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:29.464225 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.433398    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:29.529199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.497833    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:29.529199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.497938    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:29.529199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.506544    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:56:29.529199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.507305    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:56:29.529199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.507605    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" podUID="eede0436-4013-49f2-bbf6-00dade948787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/ready\": context deadline exceeded"
Nov 03 19:56:29.529199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.509841    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" podUID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.194:8000/ready\": context deadline exceeded"
Nov 03 19:56:29.556211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:29.553416    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:30.003432 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.003392    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:30.006237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.006190    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:56:30.028069 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.028021    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:30.062027 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.061994    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:56:30.081767 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.073504    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:30.093323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.091824    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:56:30.093323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.089383    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:30.114908 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.114879    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:56:30.137383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.137335    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:30.139141 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.139015    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:56:30.180933 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.180897    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:30.264104 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.264020    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:30.265272 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.265251    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 19:56:30.626246 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.626155    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"]
Nov 03 19:56:30.652839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.652806    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"]
Nov 03 19:56:30.759250 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.759204    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:30.961229 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:30.956280    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:31.912425 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:31.912377    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:32.161962 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:32.161931    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:56:32.189276 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:32.189145    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 19:56:32.192481 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:32.192457    4531 kubelet.go:1689] "Image garbage collection succeeded"
Nov 03 19:56:32.668200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:32.664796    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:32.791946 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:32.791907    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:32.804980 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:32.804952    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:33.533773 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:33.532096    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:33.731641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:33.731604    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 19:56:34.596210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:34.591303    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:34.667199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:34.666250    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 19:56:35.430759 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:35.430657    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:35.627361 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:35.627313    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb" podUID="22701240-2a28-494b-8dd7-a580ac54e26c" containerName="payload-processing" containerID="cri-o://237c607765ca1446b416080407030c6d37ffa093810b97918aed7d3ca59009e0" gracePeriod=25
Nov 03 19:56:35.654077 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:35.653996    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f" podUID="dc9f9c41-51a6-4c41-9836-d80da3895659" containerName="payload-pre-processing" containerID="cri-o://0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba" gracePeriod=25
Nov 03 19:56:35.762259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:35.762221    4531 generic.go:356] "Generic (PLEG): container finished" podID="22701240-2a28-494b-8dd7-a580ac54e26c" containerID="237c607765ca1446b416080407030c6d37ffa093810b97918aed7d3ca59009e0" exitCode=0
Nov 03 19:56:35.762399 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:35.762295    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb" event={"ID":"22701240-2a28-494b-8dd7-a580ac54e26c","Type":"ContainerDied","Data":"237c607765ca1446b416080407030c6d37ffa093810b97918aed7d3ca59009e0"}
Nov 03 19:56:35.972600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:35.972572    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:56:36.003412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.003334    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"]
Nov 03 19:56:36.004333 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.004313    4531 cpu_manager.go:414] "containerMap: removing container" podUID="22701240-2a28-494b-8dd7-a580ac54e26c" containerName="payload-processing"
Nov 03 19:56:36.004418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.004334    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22701240-2a28-494b-8dd7-a580ac54e26c" containerName="payload-processing"
Nov 03 19:56:36.004600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.004553    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="22701240-2a28-494b-8dd7-a580ac54e26c" containerName="payload-processing"
Nov 03 19:56:36.018488 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.018457    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.019445 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.019420    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"]
Nov 03 19:56:36.019554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.019489    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22701240-2a28-494b-8dd7-a580ac54e26c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22701240-2a28-494b-8dd7-a580ac54e26c-plugins-config-volume\") pod \"22701240-2a28-494b-8dd7-a580ac54e26c\" (UID: \"22701240-2a28-494b-8dd7-a580ac54e26c\") "
Nov 03 19:56:36.019554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.019534    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22701240-2a28-494b-8dd7-a580ac54e26c-kube-api-access-xpkf9\" (UniqueName: \"kubernetes.io/projected/22701240-2a28-494b-8dd7-a580ac54e26c-kube-api-access-xpkf9\") pod \"22701240-2a28-494b-8dd7-a580ac54e26c\" (UID: \"22701240-2a28-494b-8dd7-a580ac54e26c\") "
Nov 03 19:56:36.019927 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.019902    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22701240-2a28-494b-8dd7-a580ac54e26c-plugins-config-volume" pod "22701240-2a28-494b-8dd7-a580ac54e26c" (UID: "22701240-2a28-494b-8dd7-a580ac54e26c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:36.020473 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.020447    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22701240-2a28-494b-8dd7-a580ac54e26c-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:36.021234 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.021211    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d12868-openshift-default-dockercfg-vq2jt\""
Nov 03 19:56:36.022314 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.022155    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22701240-2a28-494b-8dd7-a580ac54e26c-kube-api-access-xpkf9" pod "22701240-2a28-494b-8dd7-a580ac54e26c" (UID: "22701240-2a28-494b-8dd7-a580ac54e26c"). InnerVolumeSpecName "kube-api-access-xpkf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:36.033693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.033664    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:56:36.122030 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.121991    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9f9c41-51a6-4c41-9836-d80da3895659-kube-api-access-r5n4k\" (UniqueName: \"kubernetes.io/projected/dc9f9c41-51a6-4c41-9836-d80da3895659-kube-api-access-r5n4k\") pod \"dc9f9c41-51a6-4c41-9836-d80da3895659\" (UID: \"dc9f9c41-51a6-4c41-9836-d80da3895659\") "
Nov 03 19:56:36.122480 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.122460    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc9f9c41-51a6-4c41-9836-d80da3895659-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9f9c41-51a6-4c41-9836-d80da3895659-plugins-config-volume\") pod \"dc9f9c41-51a6-4c41-9836-d80da3895659\" (UID: \"dc9f9c41-51a6-4c41-9836-d80da3895659\") "
Nov 03 19:56:36.122724 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.122698    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5fk\" (UniqueName: \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-kube-api-access-hr5fk\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.122863 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.122735    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-certs\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.122863 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.122773    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b9db67-2b47-454f-8e19-d892508e1b7c-istiod-ca-cert\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.122863 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.122814    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9f9c41-51a6-4c41-9836-d80da3895659-plugins-config-volume" pod "dc9f9c41-51a6-4c41-9836-d80da3895659" (UID: "dc9f9c41-51a6-4c41-9836-d80da3895659"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:36.123039 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.122933    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.123039 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.122960    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-envoy\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.123146 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.123078    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-credential-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.123146 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.123110    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-token\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.123307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.123160    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-data\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.123416 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.123399    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-podinfo\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.123571 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.123559    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9f9c41-51a6-4c41-9836-d80da3895659-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:36.123629 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.123575    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpkf9\" (UniqueName: \"kubernetes.io/projected/22701240-2a28-494b-8dd7-a580ac54e26c-kube-api-access-xpkf9\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:36.124302 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.124279    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9f9c41-51a6-4c41-9836-d80da3895659-kube-api-access-r5n4k" pod "dc9f9c41-51a6-4c41-9836-d80da3895659" (UID: "dc9f9c41-51a6-4c41-9836-d80da3895659"). InnerVolumeSpecName "kube-api-access-r5n4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:36.220498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.220458    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.230431    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-credential-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.230528    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-token\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.230588    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-data\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.230976    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-podinfo\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.231135    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5fk\" (UniqueName: \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-kube-api-access-hr5fk\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.231166    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-certs\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.231242    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b9db67-2b47-454f-8e19-d892508e1b7c-istiod-ca-cert\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.231338    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.231373    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-envoy\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.231558    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5n4k\" (UniqueName: \"kubernetes.io/projected/dc9f9c41-51a6-4c41-9836-d80da3895659-kube-api-access-r5n4k\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.240393    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b9db67-2b47-454f-8e19-d892508e1b7c-istiod-ca-cert\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.243197    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-credential-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.243588    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-data\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.243842    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.244152    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-certs\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.249240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.247386    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-envoy\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.261276 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.259410    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-podinfo\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.288285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.286481    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5fk\" (UniqueName: \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-kube-api-access-hr5fk\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.310128 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.310087    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-token\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.349619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.349575    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:36.642553 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:56:36.642521    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b9db67_2b47_454f_8e19_d892508e1b7c.slice/crio-d54dfae83bd5503118898c036d565a25e5b0c62a17d44755015348726b724b71 WatchSource:0}: Error finding container d54dfae83bd5503118898c036d565a25e5b0c62a17d44755015348726b724b71: Status 404 returned error can't find the container with id d54dfae83bd5503118898c036d565a25e5b0c62a17d44755015348726b724b71
Nov 03 19:56:36.645539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.645510    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"]
Nov 03 19:56:36.645702 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.645577    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:56:36.645702 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.645648    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:56:36.645702 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.645689    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:56:36.774540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.774507    4531 generic.go:356] "Generic (PLEG): container finished" podID="dc9f9c41-51a6-4c41-9836-d80da3895659" containerID="0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba" exitCode=0
Nov 03 19:56:36.774695 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.774593    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"
Nov 03 19:56:36.774750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.774596    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f" event={"ID":"dc9f9c41-51a6-4c41-9836-d80da3895659","Type":"ContainerDied","Data":"0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba"}
Nov 03 19:56:36.774750 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.774742    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f" event={"ID":"dc9f9c41-51a6-4c41-9836-d80da3895659","Type":"ContainerDied","Data":"2fe5e04e381a34c7d3cf8e58d58089392c7ff96197beeed962ec17a74b0b0e04"}
Nov 03 19:56:36.774858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.774769    4531 scope.go:122] "RemoveContainer" containerID="0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba"
Nov 03 19:56:36.776460 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.776434    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"
Nov 03 19:56:36.776596 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.776473    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb" event={"ID":"22701240-2a28-494b-8dd7-a580ac54e26c","Type":"ContainerDied","Data":"1745981a5efa0474addff1324e6cdd65109fe42c558d705c6262155a80e9714b"}
Nov 03 19:56:36.779220 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.779191    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" event={"ID":"96b9db67-2b47-454f-8e19-d892508e1b7c","Type":"ContainerStarted","Data":"f07f96380fc76cc384ce45bda54665d97d04862c90f568d2747dd310d7268057"}
Nov 03 19:56:36.779351 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.779246    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" event={"ID":"96b9db67-2b47-454f-8e19-d892508e1b7c","Type":"ContainerStarted","Data":"d54dfae83bd5503118898c036d565a25e5b0c62a17d44755015348726b724b71"}
Nov 03 19:56:36.791931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.791900    4531 scope.go:122] "RemoveContainer" containerID="0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba"
Nov 03 19:56:36.792305 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:56:36.792278    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba\": container with ID starting with 0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba not found: ID does not exist" containerID="0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba"
Nov 03 19:56:36.792401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.792315    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba"} err="failed to get container status \"0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba\": rpc error: code = NotFound desc = could not find container \"0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba\": container with ID starting with 0089bd4289334bef3c2c712ba6a8401c03c001a4388079ed1d4e15c56b9a0cba not found: ID does not exist"
Nov 03 19:56:36.792401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.792338    4531 scope.go:122] "RemoveContainer" containerID="237c607765ca1446b416080407030c6d37ffa093810b97918aed7d3ca59009e0"
Nov 03 19:56:36.801005 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.800947    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" podStartSLOduration=7.80092989 podStartE2EDuration="7.80092989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:36.797978661 +0000 UTC m=+1510.068101958" watchObservedRunningTime="2026-11-03 19:56:36.80092989 +0000 UTC m=+1510.071053190"
Nov 03 19:56:36.819283 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.819251    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"]
Nov 03 19:56:36.822125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.822098    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d0987-bd5c6ddf5-lxm9f"]
Nov 03 19:56:36.836054 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.833588    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"]
Nov 03 19:56:36.840373 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.840345    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d0987-9ffc79b6f-8hwjb"]
Nov 03 19:56:36.878619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.878579    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22701240-2a28-494b-8dd7-a580ac54e26c" path="/var/lib/kubelet/pods/22701240-2a28-494b-8dd7-a580ac54e26c/volumes"
Nov 03 19:56:36.879339 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.879316    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9f9c41-51a6-4c41-9836-d80da3895659" path="/var/lib/kubelet/pods/dc9f9c41-51a6-4c41-9836-d80da3895659/volumes"
Nov 03 19:56:36.897211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:36.897103    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:37.350217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:37.350097    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:37.356402 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:37.356377    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:37.759202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:37.757705    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:37.843197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:37.837766    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:37.843197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:37.840638    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:56:38.347777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:38.347742    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:38.764463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:38.764425    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:39.091852 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:39.091761    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:39.234322 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:39.234283    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:39.847362 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:39.847326    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:40.293168 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:40.293132    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:40.792198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:40.789319    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:41.166908 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:41.166869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:41.942034 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:41.941994    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:42.642499 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:42.642465    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:43.447200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:43.436945    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:44.156167 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:44.156094    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:45.409201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:45.406390    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:46.373195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:46.364744    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:47.755285 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:47.755208    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:48.615598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:48.615561    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:49.363829 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:49.363793    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:50.029509 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:50.029477    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:50.178199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:50.170493    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:50.537510 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:50.537461    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:50.923758 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:56:50.923715    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:56:51.631831 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:51.631795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:51.775725 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:51.775686    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:52.725825 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:52.725777    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:53.375203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:53.374858    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:54.183096 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:54.183051    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:54.346722 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:54.346660    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:54.619693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:54.619592    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:54.823395 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:54.823362    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:55.075427 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:55.075393    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:55.288197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:55.278053    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:56.010537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:56.010504    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:56.149084 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:56.149015    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:57.018853 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:57.018808    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:57.808206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:57.808033    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:58.503244 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:58.503206    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:59.240404 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:59.240376    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:59.928952 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:56:59.928904    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:00.638576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:00.638531    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:01.477841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:01.477791    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:02.128102 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:02.128067    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:02.265195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:02.264289    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:03.121619 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:03.121572    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:04.120328 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:04.120283    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:04.763083 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:04.763046    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:05.368819 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:05.368775    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:06.082197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:06.076959    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:06.751070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:06.750972    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:07.273827 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:07.273792    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:07.454563 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:07.454524    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:08.224469 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:08.224424    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:08.931358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:08.931311    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:09.637433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:09.637391    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:10.468313 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:10.468275    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:11.078160 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:11.078125    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:11.938349 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:11.938303    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:12.541313 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:12.541279    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:12.678839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:12.678805    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:12.828253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:12.822986    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:13.244582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:13.244535    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:13.425238 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:13.422232    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:14.038277 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:14.038238    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:14.751237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:14.751200    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:15.527951 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:15.527915    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:15.677910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:15.677878    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:15.891665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:15.891624    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:16.467368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:16.467328    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:16.622145 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:16.622103    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:17.349078 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:17.349005    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:17.972083 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:17.972036    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:18.728622 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:18.728571    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:19.382500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:19.382463    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:20.110003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:20.109864    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:20.255947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:20.255871    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:20.580094 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:57:20.580039    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:57:20.610196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:20.595080    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:21.030133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:21.030075    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:21.389214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:21.389153    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:21.662129 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:21.662041    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:21.816702 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:21.816671    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:21.968343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:21.968254    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:22.785195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:22.781888    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:23.718598 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:23.718563    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:24.202918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:24.202396    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:25.385465 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:25.385427    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:26.259930 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:26.259892    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:26.882308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:26.882268    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:27.668560 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:27.662386    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:28.310197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:28.309093    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:28.461697 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:28.461663    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:29.509838 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:29.509796    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:29.818502 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:29.818417    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:30.015197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:30.015090    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:30.161200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:30.157480    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:31.048199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.047934    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.071217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.067250    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.099198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.098925    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.121348 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.120270    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.160186 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.157307    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.160186 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.157764    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:57:31.160186 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.159202    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" podUID="eede0436-4013-49f2-bbf6-00dade948787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/ready\": context deadline exceeded"
Nov 03 19:57:31.190681 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.189276    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.215196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.211233    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:57:31.285440 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.285392    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.479210 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.476559    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" podUID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.548496 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.548456    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 19:57:31.550232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.550203    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:31.571309 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.571246    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl" podUID="cd542f86-cf28-4141-b7f7-b0661226c455" containerName="main" probeResult="failure" output="Get \"https://10.133.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.646540 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.646267    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.778415 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.778308    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld" podUID="4a54f455-59f9-410c-8b6a-8c56f0828800" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.839203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.838395    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.872286 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.872241    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:31.881979 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:31.881942    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:57:33.135849 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:33.135803    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:33.180610 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:33.180571    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:33.430319 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:33.353721    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:33.909199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:33.908569    4531 scope.go:122] "RemoveContainer" containerID="3c62da78e9bc3e5f41277f516a8ddcadfffbccfb79573bc22341442ff15d9b9c"
Nov 03 19:57:33.916200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:33.911356    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:35.007186 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:35.007136    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:36.244209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:36.240735    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:37.359217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:37.359160    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:38.257841 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:38.257754    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:39.293197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:39.267581    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:39.994199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:39.982996    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:40.764808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:40.764763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:41.559070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:41.559030    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:41.882380 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:41.882342    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:42.022693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:42.022653    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:42.222055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:42.221972    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:42.795346 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:42.795310    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:43.674923 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:43.674824    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:43.903740 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:43.903693    4531 generic.go:356] "Generic (PLEG): container finished" podID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerID="3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614" exitCode=0
Nov 03 19:57:43.903919 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:43.903760    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" event={"ID":"bd6ef0ea-9769-4121-8f44-39adf7de2e3b","Type":"ContainerDied","Data":"3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614"}
Nov 03 19:57:43.904406 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:43.904381    4531 scope.go:122] "RemoveContainer" containerID="3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614"
Nov 03 19:57:43.907242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:43.907156    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:43.907367 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:43.907298    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:43.907412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:43.907361    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:44.278925 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:44.278795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:44.424409 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:44.424375    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:44.781198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:44.781132    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:44.915233 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:44.915196    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" event={"ID":"bd6ef0ea-9769-4121-8f44-39adf7de2e3b","Type":"ContainerStarted","Data":"e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de"}
Nov 03 19:57:45.564401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:45.564346    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:45.609311 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:45.608755    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:57:45.609311 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:45.608992    4531 patch_prober.go:29] interesting pod/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 19:57:45.609311 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:45.609043    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 19:57:45.847255 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:45.845725    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:46.605613 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:46.605567    4531 patch_prober.go:29] interesting pod/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 19:57:46.605855 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:46.605637    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 19:57:46.671784 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:46.671706    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:47.604581 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:47.604539    4531 patch_prober.go:29] interesting pod/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 19:57:47.604985 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:47.604606    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 19:57:47.623514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:47.623481    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:48.103514 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.103483    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"]
Nov 03 19:57:48.104554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.104530    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dc9f9c41-51a6-4c41-9836-d80da3895659" containerName="payload-pre-processing"
Nov 03 19:57:48.104554 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.104553    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9f9c41-51a6-4c41-9836-d80da3895659" containerName="payload-pre-processing"
Nov 03 19:57:48.104715 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.104706    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9f9c41-51a6-4c41-9836-d80da3895659" containerName="payload-pre-processing"
Nov 03 19:57:48.109969 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.109943    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.113534 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.113504    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"]
Nov 03 19:57:48.113897 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.113877    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d12868\"/\"default-dockercfg-l7ckx\""
Nov 03 19:57:48.114379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.114355    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d12868\"/\"kube-root-ca.crt\""
Nov 03 19:57:48.114674 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.114652    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d12868\"/\"openshift-service-ca.crt\""
Nov 03 19:57:48.114788 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.114771    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d12868\"/\"test-model-d12868-kserve-self-signed-certs\""
Nov 03 19:57:48.175437 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.175399    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-home\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.175823 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.175802    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp84s\" (UniqueName: \"kubernetes.io/projected/1d818361-422a-4eb2-a643-08dc21861021-kube-api-access-zp84s\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.175958 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.175944    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d818361-422a-4eb2-a643-08dc21861021-tls-certs\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.176131 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.176120    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-model-cache\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.176232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.176220    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-dshm\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.176308 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.176300    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-tmp-dir\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.280718 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.280666    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp84s\" (UniqueName: \"kubernetes.io/projected/1d818361-422a-4eb2-a643-08dc21861021-kube-api-access-zp84s\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.293769 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.293737    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d818361-422a-4eb2-a643-08dc21861021-tls-certs\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.318360 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.318324    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-model-cache\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.319008 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.318986    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-dshm\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.319505 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.319486    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-tmp-dir\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.320458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.320438    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-home\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.320625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.318924    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-model-cache\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.320737 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.319882    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-tmp-dir\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.320834 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.320578    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-home\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.332687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.332662    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-dshm\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.333517 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.333495    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp84s\" (UniqueName: \"kubernetes.io/projected/1d818361-422a-4eb2-a643-08dc21861021-kube-api-access-zp84s\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.363272 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.360992    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d818361-422a-4eb2-a643-08dc21861021-tls-certs\") pod \"test-model-d12868-kserve-5db7944f69-fsrfl\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") " pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.395196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.395141    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:48.458201 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.454791    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:48.604870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.604827    4531 patch_prober.go:29] interesting pod/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 19:57:48.605368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.604892    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 19:57:48.835008 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:57:48.834974    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d818361_422a_4eb2_a643_08dc21861021.slice/crio-b989838003a78a509dd74af66ea8bd019242c0034a9bf942e1d35b6e84987681 WatchSource:0}: Error finding container b989838003a78a509dd74af66ea8bd019242c0034a9bf942e1d35b6e84987681: Status 404 returned error can't find the container with id b989838003a78a509dd74af66ea8bd019242c0034a9bf942e1d35b6e84987681
Nov 03 19:57:48.835830 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.835806    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"]
Nov 03 19:57:48.934288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:48.920806    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:49.056742 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:49.056703    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl" event={"ID":"1d818361-422a-4eb2-a643-08dc21861021","Type":"ContainerStarted","Data":"b989838003a78a509dd74af66ea8bd019242c0034a9bf942e1d35b6e84987681"}
Nov 03 19:57:49.604204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:49.604142    4531 patch_prober.go:29] interesting pod/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 19:57:49.604379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:49.604216    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 19:57:49.827730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:49.827688    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:50.073380 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:50.073343    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl" event={"ID":"1d818361-422a-4eb2-a643-08dc21861021","Type":"ContainerStarted","Data":"42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0"}
Nov 03 19:57:50.089072 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:50.089023    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl" podStartSLOduration=2.089008602 podStartE2EDuration="2.089008602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:50.087158647 +0000 UTC m=+1583.357281943" watchObservedRunningTime="2026-11-03 19:57:50.089008602 +0000 UTC m=+1583.359131920"
Nov 03 19:57:50.460199 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:50.456660    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:50.588700 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:57:50.588662    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:57:50.604130 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:50.604092    4531 patch_prober.go:29] interesting pod/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 19:57:50.604304 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:50.604183    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 19:57:50.980793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:50.980759    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:51.609561 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:51.609530    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:57:51.609882 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:51.609858    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:57:51.610618 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:51.610601    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 19:57:52.183887 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:52.183849    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:53.059263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:53.058281    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:53.966698 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:53.966655    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:54.697684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:54.697655    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:54.902196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:54.888404    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:55.603838 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:55.603791    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:56.495052 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:56.495017    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:57.363847 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:57.363812    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:57.497717 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:57.497686    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:57.626928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:57.626851    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:58.190357 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:58.190322    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:58.458936 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:58.458855    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:58.624105 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:58.624074    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:59.003039 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:59.003004    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:59.162207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:59.160557    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:59.245721 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:59.245682    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 19:57:59.903292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:57:59.903242    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:00.754203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:00.751759    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:01.690205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:01.689773    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:02.341195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:02.335582    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:04.254547 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:04.254518    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:04.381710 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:04.381680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:04.754422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:04.754382    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:05.375309 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.372788    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:05.517356 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.517322    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"]
Nov 03 19:58:05.526892 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.526864    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.529531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.529497    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"]
Nov 03 19:58:05.532698 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.531807    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c18bea\"/\"openshift-service-ca.crt\""
Nov 03 19:58:05.533978 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.533936    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c18bea\"/\"default-dockercfg-7hs84\""
Nov 03 19:58:05.534972 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.534360    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c18bea\"/\"test-model-c18bea-kserve-self-signed-certs\""
Nov 03 19:58:05.534972 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.534474    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c18bea\"/\"kube-root-ca.crt\""
Nov 03 19:58:05.596133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.596104    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:05.605643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.605618    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tmp-dir\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.605791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.605683    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9tg7\" (UniqueName: \"kubernetes.io/projected/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-kube-api-access-l9tg7\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.605918 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.605897    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-home\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.606061 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.605940    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tls-certs\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.606163 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.606058    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-model-cache\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.606283 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.606258    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-dshm\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.707777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.707745    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-home\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.707777 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.707783    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tls-certs\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.708011 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.707829    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-model-cache\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.708011 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.707910    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-dshm\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.708130 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.708074    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tmp-dir\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.708232 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.708159    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9tg7\" (UniqueName: \"kubernetes.io/projected/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-kube-api-access-l9tg7\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.708326 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.708230    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-home\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.708423 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.708402    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-model-cache\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.708550 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.708533    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tmp-dir\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.710263 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.710243    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-dshm\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.710582 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.710562    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tls-certs\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.714878 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.714845    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9tg7\" (UniqueName: \"kubernetes.io/projected/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-kube-api-access-l9tg7\") pod \"test-model-c18bea-kserve-9cd6df8f5-ts6w2\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") " pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:05.854196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:05.853518    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:06.229732 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:06.229700    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:06.355055 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:58:06.355025    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13a23eb6_a465_4311_8dc1_3b6cc3d0571a.slice/crio-6f9e8c23da2f3c1b7dc6cf6858120ac7797a65f384cef224d69f5b51d47b346e WatchSource:0}: Error finding container 6f9e8c23da2f3c1b7dc6cf6858120ac7797a65f384cef224d69f5b51d47b346e: Status 404 returned error can't find the container with id 6f9e8c23da2f3c1b7dc6cf6858120ac7797a65f384cef224d69f5b51d47b346e
Nov 03 19:58:06.356368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:06.356341    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"]
Nov 03 19:58:06.945138 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:06.942779    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:07.358630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:07.358547    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2" event={"ID":"13a23eb6-a465-4311-8dc1-3b6cc3d0571a","Type":"ContainerStarted","Data":"5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56"}
Nov 03 19:58:07.358630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:07.358598    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2" event={"ID":"13a23eb6-a465-4311-8dc1-3b6cc3d0571a","Type":"ContainerStarted","Data":"6f9e8c23da2f3c1b7dc6cf6858120ac7797a65f384cef224d69f5b51d47b346e"}
Nov 03 19:58:07.379094 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:07.379045    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2" podStartSLOduration=2.379031899 podStartE2EDuration="2.379031899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:07.372670568 +0000 UTC m=+1600.642793860" watchObservedRunningTime="2026-11-03 19:58:07.379031899 +0000 UTC m=+1600.649155229"
Nov 03 19:58:07.604396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:07.588654    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:07.854665 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:07.854638    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:08.547208 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:08.544101    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:09.130016 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:09.129978    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:10.143772 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:10.140029    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:10.770217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:10.756603    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:11.427228 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:11.423740    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:12.206020 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:12.193778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:12.889802 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:12.885792    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:13.592203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:13.589022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:14.423211 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:14.417818    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:15.048196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:15.042217    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:15.853196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:15.851540    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:15.857922 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:15.857139    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:15.929574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:15.929533    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:16.438719 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:16.438685    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:16.535687 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:16.535659    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 19:58:16.584630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:16.584596    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:17.118240 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:17.117432    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:17.809092 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:17.809062    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:18.438920 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:18.438881    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:19.143205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:19.132766    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:19.855288 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:19.855255    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:20.465650 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:20.465617    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:20.584845 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:58:20.584811    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:58:21.169625 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:21.169555    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:21.735638 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:21.735604    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:22.454801 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:22.454751    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:23.125855 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:23.125820    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:23.725097 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:23.725057    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:24.435891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:24.435807    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:25.126197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:25.114795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:25.794207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:25.793018    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:26.454900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:26.454850    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:27.327815 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:27.327777    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:27.557024 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:27.556988    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"]
Nov 03 19:58:27.557557 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:27.557266    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy" containerID="cri-o://9155eb75af231eefdc7b451ab1c9a3624fabd9153b7e02d1e80d4a204d90e4c5" gracePeriod=30
Nov 03 19:58:27.952682 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:27.952224    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:28.606537 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:28.606488    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:29.323347 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:29.323296    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:30.033600 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:30.033565    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:30.807478 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:30.807439    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:31.505387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:31.505351    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:32.113333 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:32.113300    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:32.727662 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:32.727628    4531 generic.go:356] "Generic (PLEG): container finished" podID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerID="9155eb75af231eefdc7b451ab1c9a3624fabd9153b7e02d1e80d4a204d90e4c5" exitCode=0
Nov 03 19:58:32.728095 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:32.727704    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" event={"ID":"385aa4c7-24cc-4a11-a69b-e572797c115f","Type":"ContainerDied","Data":"9155eb75af231eefdc7b451ab1c9a3624fabd9153b7e02d1e80d4a204d90e4c5"}
Nov 03 19:58:32.728095 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:32.727772    4531 scope.go:122] "RemoveContainer" containerID="a9750e8a6fd96da631ad0f22a2e3f730b213834905cd3fe34b63fab28b7e2e91"
Nov 03 19:58:32.928707 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:32.911779    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:32.990987 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:32.990959    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:58:33.025636 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.025600    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"]
Nov 03 19:58:33.027111 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.027086    4531 cpu_manager.go:414] "containerMap: removing container" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy"
Nov 03 19:58:33.027279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.027113    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy"
Nov 03 19:58:33.027279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.027187    4531 cpu_manager.go:414] "containerMap: removing container" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy"
Nov 03 19:58:33.027279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.027197    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy"
Nov 03 19:58:33.027624 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.027598    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy"
Nov 03 19:58:33.027624 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.027624    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" containerName="istio-proxy"
Nov 03 19:58:33.035100 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.034044    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.039414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.038442    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"]
Nov 03 19:58:33.039691 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.039639    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-11c1a8-openshift-default-dockercfg-7l2db\""
Nov 03 19:58:33.093387 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.093352    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-credential-socket\") pod \"385aa4c7-24cc-4a11-a69b-e572797c115f\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") "
Nov 03 19:58:33.093585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.093412    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-socket\") pod \"385aa4c7-24cc-4a11-a69b-e572797c115f\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") "
Nov 03 19:58:33.093585 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.093454    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-podinfo\") pod \"385aa4c7-24cc-4a11-a69b-e572797c115f\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") "
Nov 03 19:58:33.093714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.093668    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-data\") pod \"385aa4c7-24cc-4a11-a69b-e572797c115f\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") "
Nov 03 19:58:33.093714 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.093679    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-credential-socket" pod "385aa4c7-24cc-4a11-a69b-e572797c115f" (UID: "385aa4c7-24cc-4a11-a69b-e572797c115f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:33.093839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.093715    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-envoy\") pod \"385aa4c7-24cc-4a11-a69b-e572797c115f\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") "
Nov 03 19:58:33.093839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.093778    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-socket" pod "385aa4c7-24cc-4a11-a69b-e572797c115f" (UID: "385aa4c7-24cc-4a11-a69b-e572797c115f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:33.093839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.093830    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-certs\") pod \"385aa4c7-24cc-4a11-a69b-e572797c115f\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") "
Nov 03 19:58:33.094004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.093941    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-data" pod "385aa4c7-24cc-4a11-a69b-e572797c115f" (UID: "385aa4c7-24cc-4a11-a69b-e572797c115f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:33.094004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.093991    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-token\" (UniqueName: \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-token\") pod \"385aa4c7-24cc-4a11-a69b-e572797c115f\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") "
Nov 03 19:58:33.094126 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.094090    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-kube-api-access-hwpd4\" (UniqueName: \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-kube-api-access-hwpd4\") pod \"385aa4c7-24cc-4a11-a69b-e572797c115f\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") "
Nov 03 19:58:33.094126 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.094117    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/385aa4c7-24cc-4a11-a69b-e572797c115f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385aa4c7-24cc-4a11-a69b-e572797c115f-istiod-ca-cert\") pod \"385aa4c7-24cc-4a11-a69b-e572797c115f\" (UID: \"385aa4c7-24cc-4a11-a69b-e572797c115f\") "
Nov 03 19:58:33.094259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.094230    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-certs" pod "385aa4c7-24cc-4a11-a69b-e572797c115f" (UID: "385aa4c7-24cc-4a11-a69b-e572797c115f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:33.094576 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.094548    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.094704 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.094686    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2400a16-c022-45cb-91ac-8fb743616e13-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.094779 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.094757    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.094885 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.094859    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385aa4c7-24cc-4a11-a69b-e572797c115f-istiod-ca-cert" pod "385aa4c7-24cc-4a11-a69b-e572797c115f" (UID: "385aa4c7-24cc-4a11-a69b-e572797c115f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:33.094993 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.094969    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.095073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.095041    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.095242 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.095218    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.095573 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.095552    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.095743 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.095723    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2400a16-c022-45cb-91ac-8fb743616e13-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.095863 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.095842    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzsz\" (UniqueName: \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-kube-api-access-bpzsz\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.096076 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.096057    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.096492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.096111    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.096492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.096130    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385aa4c7-24cc-4a11-a69b-e572797c115f-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.096492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.096145    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.096492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.096159    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.098230 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.098154    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-envoy" pod "385aa4c7-24cc-4a11-a69b-e572797c115f" (UID: "385aa4c7-24cc-4a11-a69b-e572797c115f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:33.099364 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.099341    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-podinfo" pod "385aa4c7-24cc-4a11-a69b-e572797c115f" (UID: "385aa4c7-24cc-4a11-a69b-e572797c115f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:58:33.099453 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.099379    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-kube-api-access-hwpd4" pod "385aa4c7-24cc-4a11-a69b-e572797c115f" (UID: "385aa4c7-24cc-4a11-a69b-e572797c115f"). InnerVolumeSpecName "kube-api-access-hwpd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:33.099453 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.099412    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-token" pod "385aa4c7-24cc-4a11-a69b-e572797c115f" (UID: "385aa4c7-24cc-4a11-a69b-e572797c115f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:33.197580 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.197477    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.197580 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.197530    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2400a16-c022-45cb-91ac-8fb743616e13-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.197843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.197582    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzsz\" (UniqueName: \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-kube-api-access-bpzsz\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.197843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.197713    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.197843 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.197784    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2400a16-c022-45cb-91ac-8fb743616e13-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.198076 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.197845    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.198076 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.197914    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.198076 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.197946    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.198076 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.198009    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.198338 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.198189    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.198338 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.198233    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.198462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.198400    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.198462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.198420    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.198462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.198435    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.198462 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.198452    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpd4\" (UniqueName: \"kubernetes.io/projected/385aa4c7-24cc-4a11-a69b-e572797c115f-kube-api-access-hwpd4\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.198658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.198477    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2400a16-c022-45cb-91ac-8fb743616e13-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.198658 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.198545    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.198726 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.198670    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.200389 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.200354    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.200664 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.200641    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2400a16-c022-45cb-91ac-8fb743616e13-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.204696 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.204668    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzsz\" (UniqueName: \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-kube-api-access-bpzsz\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.204793 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.204757    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.362762 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.362724    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:33.459248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.458044    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:33.652920 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:58:33.652883    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2400a16_c022_45cb_91ac_8fb743616e13.slice/crio-e0d0c301520c9bcd0921e0615012e846577d22cd35cb42c7d9908d76984feae7 WatchSource:0}: Error finding container e0d0c301520c9bcd0921e0615012e846577d22cd35cb42c7d9908d76984feae7: Status 404 returned error can't find the container with id e0d0c301520c9bcd0921e0615012e846577d22cd35cb42c7d9908d76984feae7
Nov 03 19:58:33.655415 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.655389    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"]
Nov 03 19:58:33.656716 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.656674    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:33.656806 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.656788    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:33.656857 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.656842    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:33.748745 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.748703    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9" event={"ID":"385aa4c7-24cc-4a11-a69b-e572797c115f","Type":"ContainerDied","Data":"ef2d829221ffb02ba2b52c0731e7dd323898bc5c1999da801777fa73c5a24d79"}
Nov 03 19:58:33.749338 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.749315    4531 scope.go:122] "RemoveContainer" containerID="9155eb75af231eefdc7b451ab1c9a3624fabd9153b7e02d1e80d4a204d90e4c5"
Nov 03 19:58:33.749928 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.749908    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"
Nov 03 19:58:33.754406 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.754370    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" event={"ID":"a2400a16-c022-45cb-91ac-8fb743616e13","Type":"ContainerStarted","Data":"e0d0c301520c9bcd0921e0615012e846577d22cd35cb42c7d9908d76984feae7"}
Nov 03 19:58:33.783044 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.783006    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"]
Nov 03 19:58:33.791610 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:33.791569    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-5z9l9"]
Nov 03 19:58:34.179206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:34.161881    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:34.775019 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:34.774984    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" event={"ID":"a2400a16-c022-45cb-91ac-8fb743616e13","Type":"ContainerStarted","Data":"8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369"}
Nov 03 19:58:34.800504 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:34.800451    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" podStartSLOduration=54.800436371 podStartE2EDuration="54.800436371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:34.791315685 +0000 UTC m=+1628.061438992" watchObservedRunningTime="2026-11-03 19:58:34.800436371 +0000 UTC m=+1628.070559666"
Nov 03 19:58:34.876358 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:34.876323    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="385aa4c7-24cc-4a11-a69b-e572797c115f" path="/var/lib/kubelet/pods/385aa4c7-24cc-4a11-a69b-e572797c115f/volumes"
Nov 03 19:58:34.974195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:34.970642    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:35.363501 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:35.363466    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:35.365091 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:35.365059    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:58:35.365237 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:35.365129    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 19:58:36.065436 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:36.063886    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:36.369248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:36.368565    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:58:36.369248 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:36.368640    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 19:58:37.030531 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:37.030486    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:37.367249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:37.363579    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:58:37.367249 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:37.363644    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 19:58:38.043828 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.043778    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.077501 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.077457    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.078104 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.078070    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 19:58:38.122696 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.122652    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.128934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.128894    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.143306 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.143260    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.147305 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.143551    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" podUID="eede0436-4013-49f2-bbf6-00dade948787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.182323 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.182274    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.224302 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.224259    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:58:38.248333 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.248275    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl" podUID="1d818361-422a-4eb2-a643-08dc21861021" containerName="main" probeResult="failure" output="Get \"https://10.133.0.212:8000/ready\": context deadline exceeded"
Nov 03 19:58:38.274866 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.273301    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.339396 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.339286    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.363204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.363084    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:58:38.363204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.363145    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 19:58:38.378886 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.378799    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:38.479239 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.479195    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" podUID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.541196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.539571    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2" podUID="13a23eb6-a465-4311-8dc1-3b6cc3d0571a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": context deadline exceeded"
Nov 03 19:58:38.544214 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.541598    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.596279 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.592351    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl" podUID="cd542f86-cf28-4141-b7f7-b0661226c455" containerName="main" probeResult="failure" output="Get \"https://10.133.0.195:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.727205 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.723258    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld" podUID="4a54f455-59f9-410c-8b6a-8c56f0828800" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.769207 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.767938    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.820539 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.820311    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.848307 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.844068    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.870272 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.865168    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:38.915011 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.914968    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:58:38.941298 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:38.940994    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" podUID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.194:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:39.061198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.061132    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:39.068218 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.061771    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:58:39.093219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.080266    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:39.093219 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.080907    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:58:39.104252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.095746    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:39.104252 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.096278    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:58:39.149217 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.144250    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:39.206216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.182453    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:58:39.206656 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.206632    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 19:58:39.206925 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.206760    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 19:58:39.211193 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.209956    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8" podUID="3b05d47f-88b5-45ce-9af3-cbde59ec2d69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:39.218343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.213275    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:58:39.218343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.213356    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 19:58:39.218343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.217296    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:58:39.218343 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.215786    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" podUID="eede0436-4013-49f2-bbf6-00dade948787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:39.224693 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.224657    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:39.233117 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.233077    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:58:39.233359 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.233139    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:58:39.331614 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.331144    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 19:58:39.357276 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.354869    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 19:58:39.385957 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.385922    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:58:39.411401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.410211    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 19:58:39.411401 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.402717    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 19:58:39.426914 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.423777    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 19:58:39.535615 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.535567    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" podUID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.194:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:39.627245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.599661    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:58:39.627245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.600468    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 19:58:39.627245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.601818    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8"
Nov 03 19:58:39.635454 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:39.635411    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 19:58:40.162197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:40.157715    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:40.370254 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:40.370217    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:58:40.370568 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:40.370522    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 19:58:41.016104 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:41.016074    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:41.172709 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:41.172676    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:41.386597 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:41.386563    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:58:41.386900 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:41.386868    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 19:58:41.985125 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:41.985022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:42.363577 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:42.363485    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:58:42.364112 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:42.364082    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 19:58:43.298672 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:43.298611    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:43.380751 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:43.380719    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:43.383368 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:43.383345    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:43.391552 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:43.391524    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 19:58:44.829662 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:44.823416    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:46.121463 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:46.121420    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:46.936903 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:46.936861    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:47.631365 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:47.631331    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:48.447789 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:48.447756    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:49.034485 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:49.034413    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"]
Nov 03 19:58:49.035103 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:49.034623    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy" containerID="cri-o://b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8" gracePeriod=30
Nov 03 19:58:49.285887 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:49.285847    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:49.325913 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:49.325876    4531 generic.go:356] "Generic (PLEG): container finished" podID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerID="f07f96380fc76cc384ce45bda54665d97d04862c90f568d2747dd310d7268057" exitCode=0
Nov 03 19:58:49.331379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:49.326452    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" event={"ID":"96b9db67-2b47-454f-8e19-d892508e1b7c","Type":"ContainerDied","Data":"f07f96380fc76cc384ce45bda54665d97d04862c90f568d2747dd310d7268057"}
Nov 03 19:58:49.331379 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:49.327121    4531 scope.go:122] "RemoveContainer" containerID="f07f96380fc76cc384ce45bda54665d97d04862c90f568d2747dd310d7268057"
Nov 03 19:58:49.342341 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:49.342294    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:49.342641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:49.342406    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:49.342641 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:49.342458    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:50.046212 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:50.043624    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:50.340253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:50.338923    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" event={"ID":"96b9db67-2b47-454f-8e19-d892508e1b7c","Type":"ContainerStarted","Data":"07339454dd278a19d4fc64a22b8d56760f847d90a49e8eb4a3bd13531ac2243f"}
Nov 03 19:58:50.362197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:50.351592    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:58:50.362197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:50.351727    4531 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 19:58:50.362197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:50.351784    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 19:58:50.588607 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:58:50.588570    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:58:50.812155 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:50.812120    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:51.364245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:51.355876    4531 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 19:58:51.364245 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:51.355948    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 19:58:51.617406 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:51.615238    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:52.353748 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:52.353709    4531 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 19:58:52.354055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:52.354020    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 19:58:52.469494 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:52.469444    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:52.840455 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:52.840419    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:58:53.065197 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:53.060717    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:53.352573 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:53.352533    4531 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 19:58:53.352911 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:53.352871    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 19:58:53.832898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:53.832860    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:54.360730 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.360641    4531 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 19:58:54.361020 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.360992    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 19:58:54.412390 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.412363    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:58:54.480931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.480873    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"]
Nov 03 19:58:54.482821 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.482769    4531 cpu_manager.go:414] "containerMap: removing container" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy"
Nov 03 19:58:54.483010 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.482993    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy"
Nov 03 19:58:54.483442 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.483421    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerName="istio-proxy"
Nov 03 19:58:54.489267 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.489237    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-podinfo" pod "23d62963-94d7-4cf8-a1e9-2b73bb046ca7" (UID: "23d62963-94d7-4cf8-a1e9-2b73bb046ca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:58:54.490684 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.490657    4531 generic.go:356] "Generic (PLEG): container finished" podID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" containerID="b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8" exitCode=0
Nov 03 19:58:54.491055 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.491038    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"
Nov 03 19:58:54.492851 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.482868    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-podinfo\") pod \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") "
Nov 03 19:58:54.493033 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.493012    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-credential-socket\") pod \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") "
Nov 03 19:58:54.493675 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.493649    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-credential-socket" pod "23d62963-94d7-4cf8-a1e9-2b73bb046ca7" (UID: "23d62963-94d7-4cf8-a1e9-2b73bb046ca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:54.494374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.494343    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" event={"ID":"23d62963-94d7-4cf8-a1e9-2b73bb046ca7","Type":"ContainerDied","Data":"b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8"}
Nov 03 19:58:54.494471 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.494395    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp" event={"ID":"23d62963-94d7-4cf8-a1e9-2b73bb046ca7","Type":"ContainerDied","Data":"ac81d90c169754c79bd97ada88ce6f043ee759c3a2738b6b1b73bbd1f393c6f9"}
Nov 03 19:58:54.494471 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.494425    4531 scope.go:122] "RemoveContainer" containerID="b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8"
Nov 03 19:58:54.494623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.494606    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.497332 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.497307    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istiod-ca-cert\") pod \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") "
Nov 03 19:58:54.497500 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.497481    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-token\" (UniqueName: \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-token\") pod \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") "
Nov 03 19:58:54.497595 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.497578    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-kube-api-access-zjbfp\" (UniqueName: \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-kube-api-access-zjbfp\") pod \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") "
Nov 03 19:58:54.499241 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.499214    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istiod-ca-cert" pod "23d62963-94d7-4cf8-a1e9-2b73bb046ca7" (UID: "23d62963-94d7-4cf8-a1e9-2b73bb046ca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:54.500860 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.500834    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-data\") pod \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") "
Nov 03 19:58:54.500963 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.500921    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-socket\") pod \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") "
Nov 03 19:58:54.501070 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.501050    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-data" pod "23d62963-94d7-4cf8-a1e9-2b73bb046ca7" (UID: "23d62963-94d7-4cf8-a1e9-2b73bb046ca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:54.501187 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.501155    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-envoy\") pod \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") "
Nov 03 19:58:54.501292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.501275    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-socket" pod "23d62963-94d7-4cf8-a1e9-2b73bb046ca7" (UID: "23d62963-94d7-4cf8-a1e9-2b73bb046ca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:54.501292 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.501284    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-certs\") pod \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\" (UID: \"23d62963-94d7-4cf8-a1e9-2b73bb046ca7\") "
Nov 03 19:58:54.502374 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.502348    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"]
Nov 03 19:58:54.502494 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.502469    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:54.502574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.502504    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:54.502574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.502521    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:54.502574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.502535    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:54.502574 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.502549    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:54.502898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.502672    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-certs" pod "23d62963-94d7-4cf8-a1e9-2b73bb046ca7" (UID: "23d62963-94d7-4cf8-a1e9-2b73bb046ca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:54.503418 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.503366    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-token" pod "23d62963-94d7-4cf8-a1e9-2b73bb046ca7" (UID: "23d62963-94d7-4cf8-a1e9-2b73bb046ca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:54.505064 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.505039    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-envoy" pod "23d62963-94d7-4cf8-a1e9-2b73bb046ca7" (UID: "23d62963-94d7-4cf8-a1e9-2b73bb046ca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:54.508993 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.508974    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-kube-api-access-zjbfp" pod "23d62963-94d7-4cf8-a1e9-2b73bb046ca7" (UID: "23d62963-94d7-4cf8-a1e9-2b73bb046ca7"). InnerVolumeSpecName "kube-api-access-zjbfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:54.601216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.597665    4531 scope.go:122] "RemoveContainer" containerID="b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8"
Nov 03 19:58:54.601216 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:58:54.598092    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8\": container with ID starting with b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8 not found: ID does not exist" containerID="b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8"
Nov 03 19:58:54.601216 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.598130    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8"} err="failed to get container status \"b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8\": rpc error: code = NotFound desc = could not find container \"b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8\": container with ID starting with b5a5144c67e86fab4b3c780d6c78b81804935607578d922f5cc608e5c15485e8 not found: ID does not exist"
Nov 03 19:58:54.604383 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.604354    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6jwf\" (UniqueName: \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-kube-api-access-r6jwf\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.604507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.604425    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6364721-945b-4bfc-b118-cf93b16619d3-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.604507 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.604479    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.604699 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.604673    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.604808 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.604792    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.604931 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.604915    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.605004 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.604969    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.605068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.605042    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.605137 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.605084    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6364721-945b-4bfc-b118-cf93b16619d3-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.605412 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.605390    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjbfp\" (UniqueName: \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-kube-api-access-zjbfp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:54.605527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.605444    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:54.605527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.605463    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:54.605527 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.605479    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d62963-94d7-4cf8-a1e9-2b73bb046ca7-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:54.713450 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.713409    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.713655 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.713545    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.713655 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.713640    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.713779 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.713699    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.713838 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.713790    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.713838 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.713824    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6364721-945b-4bfc-b118-cf93b16619d3-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.714031 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.714010    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6jwf\" (UniqueName: \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-kube-api-access-r6jwf\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.714104 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.714088    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6364721-945b-4bfc-b118-cf93b16619d3-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.714159 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.714077    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.714159 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.714142    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.714575 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.714547    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.714738 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.714706    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.715003 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.714981    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.715809 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.715778    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6364721-945b-4bfc-b118-cf93b16619d3-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.725133 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.724863    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.725549 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.725523    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.726669 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.726650    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6jwf\" (UniqueName: \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-kube-api-access-r6jwf\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.726794 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.726650    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6364721-945b-4bfc-b118-cf93b16619d3-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:54.817689 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.817653    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"]
Nov 03 19:58:54.823073 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.823048    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-g9pzp"]
Nov 03 19:58:54.878206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.877695    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d62963-94d7-4cf8-a1e9-2b73bb046ca7" path="/var/lib/kubelet/pods/23d62963-94d7-4cf8-a1e9-2b73bb046ca7/volumes"
Nov 03 19:58:54.918858 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:54.918819    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:55.221231 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:55.187759    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:55.272810 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:55.272284    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"]
Nov 03 19:58:55.275256 ip-10-0-131-234 kubenswrapper[4531]: W1103 19:58:55.275222    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6364721_945b_4bfc_b118_cf93b16619d3.slice/crio-ce1599e9d4bb283db639d3e730289b32eb8fbbbff3c3707cbacf06240c13b381 WatchSource:0}: Error finding container ce1599e9d4bb283db639d3e730289b32eb8fbbbff3c3707cbacf06240c13b381: Status 404 returned error can't find the container with id ce1599e9d4bb283db639d3e730289b32eb8fbbbff3c3707cbacf06240c13b381
Nov 03 19:58:55.283530 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:55.283484    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:55.283649 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:55.283591    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:55.283649 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:55.283639    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:55.352140 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:55.351242    4531 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 19:58:55.352140 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:55.351312    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 19:58:55.502414 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:55.502370    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" event={"ID":"e6364721-945b-4bfc-b118-cf93b16619d3","Type":"ContainerStarted","Data":"48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc"}
Nov 03 19:58:55.502623 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:55.502433    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" event={"ID":"e6364721-945b-4bfc-b118-cf93b16619d3","Type":"ContainerStarted","Data":"ce1599e9d4bb283db639d3e730289b32eb8fbbbff3c3707cbacf06240c13b381"}
Nov 03 19:58:55.920011 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:55.919982    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:58:56.364898 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:56.364800    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:58:56.366747 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:56.366721    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 19:58:56.385433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:56.385380    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" podStartSLOduration=76.385365175 podStartE2EDuration="1m16.385365175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:55.521552717 +0000 UTC m=+1648.791676015" watchObservedRunningTime="2026-11-03 19:58:56.385365175 +0000 UTC m=+1649.655488477"
Nov 03 19:58:56.672912 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:56.672876    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:56.921256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:56.921224    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 19:58:56.921739 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:56.921279    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 19:58:57.919261 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:57.919228    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 19:58:57.919458 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:57.919293    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 19:58:58.462195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:58.454307    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:58.919870 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:58.919829    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 19:58:58.920157 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:58.920127    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 19:58:59.919291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:59.919256    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 19:58:59.919710 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:58:59.919311    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 19:59:00.063723 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:00.063650    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:00.933698 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:00.933665    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 19:59:00.934105 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:00.933719    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 19:59:01.198630 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:01.198538    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:01.920040 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:01.920004    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 19:59:01.920273 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:01.920063    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 19:59:02.053166 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:02.053131    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:02.862432 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:02.862393    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:02.981194 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:02.978891    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:59:03.002196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:03.001255    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:59:03.015642 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:03.015609    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 19:59:03.718453 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:03.718363    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:04.573200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:04.567498    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:05.300783 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:05.300743    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:06.038683 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:06.038650    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:06.926811 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:06.926763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:07.785891 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:07.785814    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:08.524116 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:08.524059    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:09.166970 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:09.166934    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:09.864085 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:09.864050    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:10.563206 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:10.557394    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:11.198159 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:11.195732    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:11.860741 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:11.860704    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:12.516627 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:12.516592    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:13.229202 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:13.225023    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:13.905947 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:13.905912    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:14.485259 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:14.484724    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:15.078068 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:15.078022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:15.771203 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:15.763889    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:16.437038 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:16.437001    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:16.610643 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:16.610613    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:17.223720 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:17.223684    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:17.928313 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:17.923920    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:18.087449 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:18.087415    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:18.242508 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:18.242432    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:18.594369 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:18.594259    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:19.305833 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:19.305792    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:20.012521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:20.012487    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:20.582120 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:59:20.582080    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:59:20.850639 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:20.850557    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:21.697944 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:21.697832    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:22.322704 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:22.322662    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:23.109610 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:23.089713    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:23.731498 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:23.731457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:24.448355 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:24.448272    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:25.126874 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:25.126838    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:25.871910 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:25.871864    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:26.623310 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:26.623248    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:27.223042 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:27.223004    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:27.934196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:27.932659    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:28.645422 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:28.642405    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:29.271435 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:29.271397    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:29.917927 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:29.917876    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:30.856315 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:30.856280    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:31.431701 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:31.431662    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:32.124934 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:32.124896    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:32.943198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:32.939199    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:33.576256 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:33.567325    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:34.064099 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:34.064066    4531 scope.go:122] "RemoveContainer" containerID="c5dac8959025797f18710a7d2efc6e85c442ca52cbdbbbe899e45eb6710e6db7"
Nov 03 19:59:34.257320 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:34.257263    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:35.028204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:35.026654    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:35.649840 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:35.649797    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:36.325091 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:36.325054    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:36.960474 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:36.960437    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:37.553766 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:37.553695    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:38.253823 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:38.253782    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:38.877280 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:38.877239    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:39.513022 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:39.512989    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:40.256491 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:40.241152    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:40.946521 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:40.946469    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:41.546312 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:41.544364    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:42.441298 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:42.441260    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:43.021433 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:43.021395    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:43.683198 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:43.675080    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:44.410196 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:44.401296    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:45.145791 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:45.145752    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:45.844222 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:45.843876    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:46.540200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:46.528620    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:47.163803 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:47.163766    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:47.952115 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:47.952059    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:48.578012 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:48.577979    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:49.344822 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:49.344775    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:50.168204 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:50.137851    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:50.577143 ip-10-0-131-234 kubenswrapper[4531]: E1103 19:59:50.577054    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 19:59:50.767200 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:50.766753    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:51.447817 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:51.447777    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:52.025873 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:52.025832    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:52.622291 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:52.622240    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:53.337350 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:53.327038    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:54.041492 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:54.041394    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:54.642296 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:54.642249    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:55.397839 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:55.397799    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:56.057517 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:56.057393    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:56.943253 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:56.938516    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:57.538086 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:57.527353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:58.238209 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:58.229870    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:58.994604 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:58.994563    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:59.610195 ip-10-0-131-234 kubenswrapper[4531]: I1103 19:59:59.603211    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:00.141782 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.141734    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"]
Nov 03 20:00:00.149255 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.149221    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"
Nov 03 20:00:00.154540 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.151876    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"]
Nov 03 20:00:00.167497 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.167469    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"
Nov 03 20:00:00.210800 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.210765    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"]
Nov 03 20:00:00.223884 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.219398    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"]
Nov 03 20:00:00.228866 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.228821    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"]
Nov 03 20:00:00.242135 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.242101    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j6gf\" (UniqueName: \"kubernetes.io/projected/169857d1-ca33-4bde-a129-84922e265d30-kube-api-access-4j6gf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb\" (UID: \"169857d1-ca33-4bde-a129-84922e265d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"
Nov 03 20:00:00.242368 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.242241    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169857d1-ca33-4bde-a129-84922e265d30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb\" (UID: \"169857d1-ca33-4bde-a129-84922e265d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"
Nov 03 20:00:00.250954 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.250919    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:00:00.251650 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.251624    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"]
Nov 03 20:00:00.251931 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.251912    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"
Nov 03 20:00:00.254601 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.254577    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"]
Nov 03 20:00:00.261146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.261125    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"
Nov 03 20:00:00.266766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.266740    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"]
Nov 03 20:00:00.343929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.343893    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgg9k\" (UniqueName: \"kubernetes.io/projected/6c661d33-3c04-4423-8138-aa741b7a0c78-kube-api-access-hgg9k\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2\" (UID: \"6c661d33-3c04-4423-8138-aa741b7a0c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"
Nov 03 20:00:00.344116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.344071    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smm5f\" (UniqueName: \"kubernetes.io/projected/ebc39eb6-3905-4203-b242-a28f6b12477a-kube-api-access-smm5f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp\" (UID: \"ebc39eb6-3905-4203-b242-a28f6b12477a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"
Nov 03 20:00:00.344218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.344203    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6gf\" (UniqueName: \"kubernetes.io/projected/169857d1-ca33-4bde-a129-84922e265d30-kube-api-access-4j6gf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb\" (UID: \"169857d1-ca33-4bde-a129-84922e265d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"
Nov 03 20:00:00.344284 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.344274    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169857d1-ca33-4bde-a129-84922e265d30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb\" (UID: \"169857d1-ca33-4bde-a129-84922e265d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"
Nov 03 20:00:00.344323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.344297    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c661d33-3c04-4423-8138-aa741b7a0c78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2\" (UID: \"6c661d33-3c04-4423-8138-aa741b7a0c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"
Nov 03 20:00:00.344405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.344393    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc39eb6-3905-4203-b242-a28f6b12477a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp\" (UID: \"ebc39eb6-3905-4203-b242-a28f6b12477a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"
Nov 03 20:00:00.345014 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.344993    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169857d1-ca33-4bde-a129-84922e265d30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb\" (UID: \"169857d1-ca33-4bde-a129-84922e265d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"
Nov 03 20:00:00.351656 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.351619    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j6gf\" (UniqueName: \"kubernetes.io/projected/169857d1-ca33-4bde-a129-84922e265d30-kube-api-access-4j6gf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb\" (UID: \"169857d1-ca33-4bde-a129-84922e265d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"
Nov 03 20:00:00.409249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.407838    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:00.467217 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.460780    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc39eb6-3905-4203-b242-a28f6b12477a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp\" (UID: \"ebc39eb6-3905-4203-b242-a28f6b12477a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"
Nov 03 20:00:00.467642 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.467611    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6w5\" (UniqueName: \"kubernetes.io/projected/10367631-3bd7-476a-88fd-bc1695606723-kube-api-access-hd6w5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj\" (UID: \"10367631-3bd7-476a-88fd-bc1695606723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"
Nov 03 20:00:00.468142 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.468108    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgg9k\" (UniqueName: \"kubernetes.io/projected/6c661d33-3c04-4423-8138-aa741b7a0c78-kube-api-access-hgg9k\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2\" (UID: \"6c661d33-3c04-4423-8138-aa741b7a0c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"
Nov 03 20:00:00.473807 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.468665    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smm5f\" (UniqueName: \"kubernetes.io/projected/ebc39eb6-3905-4203-b242-a28f6b12477a-kube-api-access-smm5f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp\" (UID: \"ebc39eb6-3905-4203-b242-a28f6b12477a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"
Nov 03 20:00:00.474569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.474530    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10367631-3bd7-476a-88fd-bc1695606723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj\" (UID: \"10367631-3bd7-476a-88fd-bc1695606723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"
Nov 03 20:00:00.474857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.474831    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c661d33-3c04-4423-8138-aa741b7a0c78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2\" (UID: \"6c661d33-3c04-4423-8138-aa741b7a0c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"
Nov 03 20:00:00.475413 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.466895    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc39eb6-3905-4203-b242-a28f6b12477a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp\" (UID: \"ebc39eb6-3905-4203-b242-a28f6b12477a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"
Nov 03 20:00:00.476626 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.476587    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c661d33-3c04-4423-8138-aa741b7a0c78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2\" (UID: \"6c661d33-3c04-4423-8138-aa741b7a0c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"
Nov 03 20:00:00.491199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.481574    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"
Nov 03 20:00:00.525074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.525038    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smm5f\" (UniqueName: \"kubernetes.io/projected/ebc39eb6-3905-4203-b242-a28f6b12477a-kube-api-access-smm5f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp\" (UID: \"ebc39eb6-3905-4203-b242-a28f6b12477a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"
Nov 03 20:00:00.525966 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.525879    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgg9k\" (UniqueName: \"kubernetes.io/projected/6c661d33-3c04-4423-8138-aa741b7a0c78-kube-api-access-hgg9k\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2\" (UID: \"6c661d33-3c04-4423-8138-aa741b7a0c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"
Nov 03 20:00:00.575268 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.574929    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"
Nov 03 20:00:00.578812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.578771    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10367631-3bd7-476a-88fd-bc1695606723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj\" (UID: \"10367631-3bd7-476a-88fd-bc1695606723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"
Nov 03 20:00:00.579407 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.579025    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6w5\" (UniqueName: \"kubernetes.io/projected/10367631-3bd7-476a-88fd-bc1695606723-kube-api-access-hd6w5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj\" (UID: \"10367631-3bd7-476a-88fd-bc1695606723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"
Nov 03 20:00:00.580606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.580159    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10367631-3bd7-476a-88fd-bc1695606723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj\" (UID: \"10367631-3bd7-476a-88fd-bc1695606723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"
Nov 03 20:00:00.586894 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.586864    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6w5\" (UniqueName: \"kubernetes.io/projected/10367631-3bd7-476a-88fd-bc1695606723-kube-api-access-hd6w5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj\" (UID: \"10367631-3bd7-476a-88fd-bc1695606723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"
Nov 03 20:00:00.592778 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.592751    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"
Nov 03 20:00:00.801206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.801154    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"
Nov 03 20:00:00.815862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.815822    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"]
Nov 03 20:00:00.818362 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:00:00.818332    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169857d1_ca33_4bde_a129_84922e265d30.slice/crio-75397edf05d3e0d664f9ac8e9ba47ca5e2d197107af59df1cdf3512494673ee3 WatchSource:0}: Error finding container 75397edf05d3e0d664f9ac8e9ba47ca5e2d197107af59df1cdf3512494673ee3: Status 404 returned error can't find the container with id 75397edf05d3e0d664f9ac8e9ba47ca5e2d197107af59df1cdf3512494673ee3
Nov 03 20:00:00.909433 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:00:00.909358    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc39eb6_3905_4203_b242_a28f6b12477a.slice/crio-3c8d3329d32837057b6fd16c942c8c7dba0748946940620c3b5981cd9c18497b WatchSource:0}: Error finding container 3c8d3329d32837057b6fd16c942c8c7dba0748946940620c3b5981cd9c18497b: Status 404 returned error can't find the container with id 3c8d3329d32837057b6fd16c942c8c7dba0748946940620c3b5981cd9c18497b
Nov 03 20:00:00.913164 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.913131    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"]
Nov 03 20:00:00.934664 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:00.934632    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"]
Nov 03 20:00:00.935968 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:00:00.935938    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10367631_3bd7_476a_88fd_bc1695606723.slice/crio-fab54b704770ee9b6d01bc4deff34e92f541827675e33a33733427fe2451fb3a WatchSource:0}: Error finding container fab54b704770ee9b6d01bc4deff34e92f541827675e33a33733427fe2451fb3a: Status 404 returned error can't find the container with id fab54b704770ee9b6d01bc4deff34e92f541827675e33a33733427fe2451fb3a
Nov 03 20:00:01.107035 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:00:01.106989    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c661d33_3c04_4423_8138_aa741b7a0c78.slice/crio-1aeb043cbf2c72f46efb3140752578019f1e10b84211e9efea063d086988ccfd WatchSource:0}: Error finding container 1aeb043cbf2c72f46efb3140752578019f1e10b84211e9efea063d086988ccfd: Status 404 returned error can't find the container with id 1aeb043cbf2c72f46efb3140752578019f1e10b84211e9efea063d086988ccfd
Nov 03 20:00:01.107162 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:01.107038    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"]
Nov 03 20:00:01.272956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:01.262438    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:01.529658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:01.529560    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp" event={"ID":"ebc39eb6-3905-4203-b242-a28f6b12477a","Type":"ContainerStarted","Data":"3c8d3329d32837057b6fd16c942c8c7dba0748946940620c3b5981cd9c18497b"}
Nov 03 20:00:01.531452 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:01.531420    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb" event={"ID":"169857d1-ca33-4bde-a129-84922e265d30","Type":"ContainerStarted","Data":"75397edf05d3e0d664f9ac8e9ba47ca5e2d197107af59df1cdf3512494673ee3"}
Nov 03 20:00:01.532968 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:01.532935    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2" event={"ID":"6c661d33-3c04-4423-8138-aa741b7a0c78","Type":"ContainerStarted","Data":"1aeb043cbf2c72f46efb3140752578019f1e10b84211e9efea063d086988ccfd"}
Nov 03 20:00:01.534637 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:01.534606    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj" event={"ID":"10367631-3bd7-476a-88fd-bc1695606723","Type":"ContainerStarted","Data":"fab54b704770ee9b6d01bc4deff34e92f541827675e33a33733427fe2451fb3a"}
Nov 03 20:00:02.071198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.060763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:02.545765 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.545731    4531 generic.go:356] "Generic (PLEG): container finished" podID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerID="d007fd23db6d8cc4ba2c90b0e8ebfae0d81094a4b26968b9901be2668d130825" exitCode=60
Nov 03 20:00:02.546262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.545808    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp" event={"ID":"ebc39eb6-3905-4203-b242-a28f6b12477a","Type":"ContainerDied","Data":"d007fd23db6d8cc4ba2c90b0e8ebfae0d81094a4b26968b9901be2668d130825"}
Nov 03 20:00:02.546355 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.546269    4531 scope.go:122] "RemoveContainer" containerID="d007fd23db6d8cc4ba2c90b0e8ebfae0d81094a4b26968b9901be2668d130825"
Nov 03 20:00:02.547948 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.547920    4531 generic.go:356] "Generic (PLEG): container finished" podID="169857d1-ca33-4bde-a129-84922e265d30" containerID="8a3ed840bd565ac9374e6ea0f3e043b55980df1a51b976d4af7b6c173865b360" exitCode=60
Nov 03 20:00:02.548076 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.547994    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb" event={"ID":"169857d1-ca33-4bde-a129-84922e265d30","Type":"ContainerDied","Data":"8a3ed840bd565ac9374e6ea0f3e043b55980df1a51b976d4af7b6c173865b360"}
Nov 03 20:00:02.548758 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.548524    4531 scope.go:122] "RemoveContainer" containerID="8a3ed840bd565ac9374e6ea0f3e043b55980df1a51b976d4af7b6c173865b360"
Nov 03 20:00:02.551149 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.551130    4531 generic.go:356] "Generic (PLEG): container finished" podID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerID="012d36f15edf95ea0312bf3abfbb9c4192af591acbe1bcb97e0bd63deb2b38e1" exitCode=60
Nov 03 20:00:02.551243 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.551223    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2" event={"ID":"6c661d33-3c04-4423-8138-aa741b7a0c78","Type":"ContainerDied","Data":"012d36f15edf95ea0312bf3abfbb9c4192af591acbe1bcb97e0bd63deb2b38e1"}
Nov 03 20:00:02.552097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.552073    4531 scope.go:122] "RemoveContainer" containerID="012d36f15edf95ea0312bf3abfbb9c4192af591acbe1bcb97e0bd63deb2b38e1"
Nov 03 20:00:02.553343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.553321    4531 generic.go:356] "Generic (PLEG): container finished" podID="10367631-3bd7-476a-88fd-bc1695606723" containerID="4fd539cb595348ae147bd6eb5227607c651437cbc5d11c41145e1ac2b7bb7394" exitCode=60
Nov 03 20:00:02.553468 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.553407    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj" event={"ID":"10367631-3bd7-476a-88fd-bc1695606723","Type":"ContainerDied","Data":"4fd539cb595348ae147bd6eb5227607c651437cbc5d11c41145e1ac2b7bb7394"}
Nov 03 20:00:02.553686 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.553666    4531 scope.go:122] "RemoveContainer" containerID="4fd539cb595348ae147bd6eb5227607c651437cbc5d11c41145e1ac2b7bb7394"
Nov 03 20:00:02.851262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:02.847697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:03.565084 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:03.564985    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb" event={"ID":"169857d1-ca33-4bde-a129-84922e265d30","Type":"ContainerStarted","Data":"11576e5c715077d00182f6dd4545760345e5f91c920df1a3c20ae2ea36613125"}
Nov 03 20:00:03.567940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:03.567907    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2" event={"ID":"6c661d33-3c04-4423-8138-aa741b7a0c78","Type":"ContainerStarted","Data":"5c39e057fcd0fef054a93b9a8edf02a7a97127568a972465e0069dc4048be933"}
Nov 03 20:00:03.570472 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:03.570442    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj" event={"ID":"10367631-3bd7-476a-88fd-bc1695606723","Type":"ContainerStarted","Data":"f84c54e6621e1c2621291d108ddda6a16aabaa4b2f4b99e6f7be0c616d95feb7"}
Nov 03 20:00:03.572649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:03.572624    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp" event={"ID":"ebc39eb6-3905-4203-b242-a28f6b12477a","Type":"ContainerStarted","Data":"5f6f7daa691c3c25322e195d73a2a4492e6e49d612f63d6e7441305700705d2d"}
Nov 03 20:00:03.581787 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:03.581741    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb" podStartSLOduration=3.581729341 podStartE2EDuration="3.581729341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:03.579516011 +0000 UTC m=+1716.849639306" watchObservedRunningTime="2026-11-03 20:00:03.581729341 +0000 UTC m=+1716.851852636"
Nov 03 20:00:03.599219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:03.599144    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp" podStartSLOduration=3.599130757 podStartE2EDuration="3.599130757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:03.595117821 +0000 UTC m=+1716.865241116" watchObservedRunningTime="2026-11-03 20:00:03.599130757 +0000 UTC m=+1716.869254087"
Nov 03 20:00:03.613588 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:03.613531    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2" podStartSLOduration=3.613517177 podStartE2EDuration="3.613517177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:03.607328971 +0000 UTC m=+1716.877452265" watchObservedRunningTime="2026-11-03 20:00:03.613517177 +0000 UTC m=+1716.883640471"
Nov 03 20:00:03.627265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:03.627208    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj" podStartSLOduration=3.627189402 podStartE2EDuration="3.627189402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:03.621886827 +0000 UTC m=+1716.892010122" watchObservedRunningTime="2026-11-03 20:00:03.627189402 +0000 UTC m=+1716.897312697"
Nov 03 20:00:03.691213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:03.682161    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:04.407213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.405767    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:04.585546 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.585512    4531 generic.go:356] "Generic (PLEG): container finished" podID="169857d1-ca33-4bde-a129-84922e265d30" containerID="11576e5c715077d00182f6dd4545760345e5f91c920df1a3c20ae2ea36613125" exitCode=60
Nov 03 20:00:04.586074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.585594    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb" event={"ID":"169857d1-ca33-4bde-a129-84922e265d30","Type":"ContainerDied","Data":"11576e5c715077d00182f6dd4545760345e5f91c920df1a3c20ae2ea36613125"}
Nov 03 20:00:04.586074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.585672    4531 scope.go:122] "RemoveContainer" containerID="8a3ed840bd565ac9374e6ea0f3e043b55980df1a51b976d4af7b6c173865b360"
Nov 03 20:00:04.586074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.586010    4531 scope.go:122] "RemoveContainer" containerID="11576e5c715077d00182f6dd4545760345e5f91c920df1a3c20ae2ea36613125"
Nov 03 20:00:04.586615 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:04.586587    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb_odh-ai-gateway-infra(169857d1-ca33-4bde-a129-84922e265d30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb" podUID="169857d1-ca33-4bde-a129-84922e265d30"
Nov 03 20:00:04.588158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.588134    4531 generic.go:356] "Generic (PLEG): container finished" podID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerID="5c39e057fcd0fef054a93b9a8edf02a7a97127568a972465e0069dc4048be933" exitCode=60
Nov 03 20:00:04.588318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.588236    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2" event={"ID":"6c661d33-3c04-4423-8138-aa741b7a0c78","Type":"ContainerDied","Data":"5c39e057fcd0fef054a93b9a8edf02a7a97127568a972465e0069dc4048be933"}
Nov 03 20:00:04.588692 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.588667    4531 scope.go:122] "RemoveContainer" containerID="5c39e057fcd0fef054a93b9a8edf02a7a97127568a972465e0069dc4048be933"
Nov 03 20:00:04.589245 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:04.589210    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2_odh-ai-gateway-infra(6c661d33-3c04-4423-8138-aa741b7a0c78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78"
Nov 03 20:00:04.590639 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.590617    4531 generic.go:356] "Generic (PLEG): container finished" podID="10367631-3bd7-476a-88fd-bc1695606723" containerID="f84c54e6621e1c2621291d108ddda6a16aabaa4b2f4b99e6f7be0c616d95feb7" exitCode=60
Nov 03 20:00:04.590755 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.590714    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj" event={"ID":"10367631-3bd7-476a-88fd-bc1695606723","Type":"ContainerDied","Data":"f84c54e6621e1c2621291d108ddda6a16aabaa4b2f4b99e6f7be0c616d95feb7"}
Nov 03 20:00:04.591002 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.590978    4531 scope.go:122] "RemoveContainer" containerID="f84c54e6621e1c2621291d108ddda6a16aabaa4b2f4b99e6f7be0c616d95feb7"
Nov 03 20:00:04.591430 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:04.591407    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj_odh-ai-gateway-infra(10367631-3bd7-476a-88fd-bc1695606723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj" podUID="10367631-3bd7-476a-88fd-bc1695606723"
Nov 03 20:00:04.593245 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.593222    4531 generic.go:356] "Generic (PLEG): container finished" podID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerID="5f6f7daa691c3c25322e195d73a2a4492e6e49d612f63d6e7441305700705d2d" exitCode=60
Nov 03 20:00:04.593333 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.593286    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp" event={"ID":"ebc39eb6-3905-4203-b242-a28f6b12477a","Type":"ContainerDied","Data":"5f6f7daa691c3c25322e195d73a2a4492e6e49d612f63d6e7441305700705d2d"}
Nov 03 20:00:04.593560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.593547    4531 scope.go:122] "RemoveContainer" containerID="5f6f7daa691c3c25322e195d73a2a4492e6e49d612f63d6e7441305700705d2d"
Nov 03 20:00:04.593844 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:04.593828    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp_odh-ai-gateway-infra(ebc39eb6-3905-4203-b242-a28f6b12477a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a"
Nov 03 20:00:04.618434 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.604287    4531 scope.go:122] "RemoveContainer" containerID="012d36f15edf95ea0312bf3abfbb9c4192af591acbe1bcb97e0bd63deb2b38e1"
Nov 03 20:00:04.638401 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.638378    4531 scope.go:122] "RemoveContainer" containerID="4fd539cb595348ae147bd6eb5227607c651437cbc5d11c41145e1ac2b7bb7394"
Nov 03 20:00:04.655859 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:04.655832    4531 scope.go:122] "RemoveContainer" containerID="d007fd23db6d8cc4ba2c90b0e8ebfae0d81094a4b26968b9901be2668d130825"
Nov 03 20:00:05.075249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:05.069416    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:05.605900 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:05.605866    4531 scope.go:122] "RemoveContainer" containerID="5f6f7daa691c3c25322e195d73a2a4492e6e49d612f63d6e7441305700705d2d"
Nov 03 20:00:05.606509 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:05.606482    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp_odh-ai-gateway-infra(ebc39eb6-3905-4203-b242-a28f6b12477a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a"
Nov 03 20:00:05.607787 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:05.607766    4531 scope.go:122] "RemoveContainer" containerID="11576e5c715077d00182f6dd4545760345e5f91c920df1a3c20ae2ea36613125"
Nov 03 20:00:05.608310 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:05.608285    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb_odh-ai-gateway-infra(169857d1-ca33-4bde-a129-84922e265d30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb" podUID="169857d1-ca33-4bde-a129-84922e265d30"
Nov 03 20:00:05.609639 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:05.609574    4531 scope.go:122] "RemoveContainer" containerID="5c39e057fcd0fef054a93b9a8edf02a7a97127568a972465e0069dc4048be933"
Nov 03 20:00:05.610163 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:05.610051    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2_odh-ai-gateway-infra(6c661d33-3c04-4423-8138-aa741b7a0c78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78"
Nov 03 20:00:05.611873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:05.611852    4531 scope.go:122] "RemoveContainer" containerID="f84c54e6621e1c2621291d108ddda6a16aabaa4b2f4b99e6f7be0c616d95feb7"
Nov 03 20:00:05.612305 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:05.612282    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj_odh-ai-gateway-infra(10367631-3bd7-476a-88fd-bc1695606723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj" podUID="10367631-3bd7-476a-88fd-bc1695606723"
Nov 03 20:00:06.025130 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:06.025089    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:06.672199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:06.671264    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:07.420769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:07.420731    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:08.140205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:08.137135    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:08.746760 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:08.746722    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:09.536266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:09.536231    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:10.273852 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:10.273814    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:11.405198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:11.404255    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" podUID="eede0436-4013-49f2-bbf6-00dade948787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/ready\": context deadline exceeded"
Nov 03 20:00:11.461256 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:11.461160    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:00:12.384233 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:12.384167    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:13.206718 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:13.206670    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:14.140723 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:14.140680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:15.013529 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:15.013439    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:15.830929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:15.830890    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:15.882306 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:15.882272    4531 scope.go:122] "RemoveContainer" containerID="5c39e057fcd0fef054a93b9a8edf02a7a97127568a972465e0069dc4048be933"
Nov 03 20:00:15.894354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:15.894318    4531 scope.go:122] "RemoveContainer" containerID="f84c54e6621e1c2621291d108ddda6a16aabaa4b2f4b99e6f7be0c616d95feb7"
Nov 03 20:00:15.895033 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:15.895005    4531 scope.go:122] "RemoveContainer" containerID="11576e5c715077d00182f6dd4545760345e5f91c920df1a3c20ae2ea36613125"
Nov 03 20:00:16.549206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:16.547298    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:16.760884 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:16.760840    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb" event={"ID":"169857d1-ca33-4bde-a129-84922e265d30","Type":"ContainerStarted","Data":"7c9a7a93a26d07a1348a91d305f3de2fe881a2a27345e306404acbcad14f983f"}
Nov 03 20:00:16.763131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:16.763101    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2" event={"ID":"6c661d33-3c04-4423-8138-aa741b7a0c78","Type":"ContainerStarted","Data":"47c5ed5af51545fbdea281aee68266a354c0cfbd4521d47b505043cc52195a29"}
Nov 03 20:00:16.765522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:16.765497    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj" event={"ID":"10367631-3bd7-476a-88fd-bc1695606723","Type":"ContainerStarted","Data":"242c2fe5fe2e1fd820a3653eb421b16219b05976e34521b52207357e76cf186c"}
Nov 03 20:00:16.771511 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:16.771489    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:16.859288 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:16.859139    4531 scope.go:122] "RemoveContainer" containerID="5f6f7daa691c3c25322e195d73a2a4492e6e49d612f63d6e7441305700705d2d"
Nov 03 20:00:17.364575 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.364516    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:17.552792 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.552711    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:17.777308 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.777253    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp" event={"ID":"ebc39eb6-3905-4203-b242-a28f6b12477a","Type":"ContainerStarted","Data":"d6c3c65cec9e41cc144956c2ca107a9647cbb335a27ba27fbc0d8be6eb89d5b1"}
Nov 03 20:00:17.780050 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.780016    4531 generic.go:356] "Generic (PLEG): container finished" podID="169857d1-ca33-4bde-a129-84922e265d30" containerID="7c9a7a93a26d07a1348a91d305f3de2fe881a2a27345e306404acbcad14f983f" exitCode=60
Nov 03 20:00:17.780050 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.780045    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb" event={"ID":"169857d1-ca33-4bde-a129-84922e265d30","Type":"ContainerDied","Data":"7c9a7a93a26d07a1348a91d305f3de2fe881a2a27345e306404acbcad14f983f"}
Nov 03 20:00:17.780372 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.780115    4531 scope.go:122] "RemoveContainer" containerID="11576e5c715077d00182f6dd4545760345e5f91c920df1a3c20ae2ea36613125"
Nov 03 20:00:17.780636 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.780614    4531 scope.go:122] "RemoveContainer" containerID="7c9a7a93a26d07a1348a91d305f3de2fe881a2a27345e306404acbcad14f983f"
Nov 03 20:00:17.781134 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:17.781107    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb_odh-ai-gateway-infra(169857d1-ca33-4bde-a129-84922e265d30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb" podUID="169857d1-ca33-4bde-a129-84922e265d30"
Nov 03 20:00:17.784498 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.784472    4531 generic.go:356] "Generic (PLEG): container finished" podID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerID="47c5ed5af51545fbdea281aee68266a354c0cfbd4521d47b505043cc52195a29" exitCode=60
Nov 03 20:00:17.784622 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.784554    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2" event={"ID":"6c661d33-3c04-4423-8138-aa741b7a0c78","Type":"ContainerDied","Data":"47c5ed5af51545fbdea281aee68266a354c0cfbd4521d47b505043cc52195a29"}
Nov 03 20:00:17.785032 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.785011    4531 scope.go:122] "RemoveContainer" containerID="47c5ed5af51545fbdea281aee68266a354c0cfbd4521d47b505043cc52195a29"
Nov 03 20:00:17.785534 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:17.785507    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2_odh-ai-gateway-infra(6c661d33-3c04-4423-8138-aa741b7a0c78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78"
Nov 03 20:00:17.787509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.787487    4531 generic.go:356] "Generic (PLEG): container finished" podID="10367631-3bd7-476a-88fd-bc1695606723" containerID="242c2fe5fe2e1fd820a3653eb421b16219b05976e34521b52207357e76cf186c" exitCode=60
Nov 03 20:00:17.787681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.787653    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj" event={"ID":"10367631-3bd7-476a-88fd-bc1695606723","Type":"ContainerDied","Data":"242c2fe5fe2e1fd820a3653eb421b16219b05976e34521b52207357e76cf186c"}
Nov 03 20:00:17.788218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.788154    4531 scope.go:122] "RemoveContainer" containerID="242c2fe5fe2e1fd820a3653eb421b16219b05976e34521b52207357e76cf186c"
Nov 03 20:00:17.788667 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:17.788631    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj_odh-ai-gateway-infra(10367631-3bd7-476a-88fd-bc1695606723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj" podUID="10367631-3bd7-476a-88fd-bc1695606723"
Nov 03 20:00:17.792257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.792158    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"]
Nov 03 20:00:17.804786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.804756    4531 scope.go:122] "RemoveContainer" containerID="5c39e057fcd0fef054a93b9a8edf02a7a97127568a972465e0069dc4048be933"
Nov 03 20:00:17.811300 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.811267    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"]
Nov 03 20:00:17.824583 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.824553    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"]
Nov 03 20:00:17.825278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:17.825257    4531 scope.go:122] "RemoveContainer" containerID="f84c54e6621e1c2621291d108ddda6a16aabaa4b2f4b99e6f7be0c616d95feb7"
Nov 03 20:00:18.313201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:18.312744    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:18.801620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:18.801577    4531 generic.go:356] "Generic (PLEG): container finished" podID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerID="d6c3c65cec9e41cc144956c2ca107a9647cbb335a27ba27fbc0d8be6eb89d5b1" exitCode=60
Nov 03 20:00:18.802011 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:18.801757    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp" event={"ID":"ebc39eb6-3905-4203-b242-a28f6b12477a","Type":"ContainerDied","Data":"d6c3c65cec9e41cc144956c2ca107a9647cbb335a27ba27fbc0d8be6eb89d5b1"}
Nov 03 20:00:18.802011 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:18.801823    4531 scope.go:122] "RemoveContainer" containerID="5f6f7daa691c3c25322e195d73a2a4492e6e49d612f63d6e7441305700705d2d"
Nov 03 20:00:18.802480 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:18.802453    4531 scope.go:122] "RemoveContainer" containerID="d6c3c65cec9e41cc144956c2ca107a9647cbb335a27ba27fbc0d8be6eb89d5b1"
Nov 03 20:00:18.803057 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:18.803022    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp_odh-ai-gateway-infra(ebc39eb6-3905-4203-b242-a28f6b12477a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a"
Nov 03 20:00:18.809896 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:18.809865    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"]
Nov 03 20:00:18.997211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:18.995553    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:19.209899 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.209864    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"
Nov 03 20:00:19.265460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.261669    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"]
Nov 03 20:00:19.265460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.265059    4531 cpu_manager.go:414] "containerMap: removing container" podUID="169857d1-ca33-4bde-a129-84922e265d30" containerName="cleanup"
Nov 03 20:00:19.265460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.265088    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169857d1-ca33-4bde-a129-84922e265d30" containerName="cleanup"
Nov 03 20:00:19.265460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.265119    4531 cpu_manager.go:414] "containerMap: removing container" podUID="169857d1-ca33-4bde-a129-84922e265d30" containerName="cleanup"
Nov 03 20:00:19.265460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.265129    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169857d1-ca33-4bde-a129-84922e265d30" containerName="cleanup"
Nov 03 20:00:19.265460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.265440    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="169857d1-ca33-4bde-a129-84922e265d30" containerName="cleanup"
Nov 03 20:00:19.265460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.265458    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="169857d1-ca33-4bde-a129-84922e265d30" containerName="cleanup"
Nov 03 20:00:19.265460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.265474    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="169857d1-ca33-4bde-a129-84922e265d30" containerName="cleanup"
Nov 03 20:00:19.276190 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.273541    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"
Nov 03 20:00:19.292656 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.292612    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"]
Nov 03 20:00:19.335431 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.335393    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169857d1-ca33-4bde-a129-84922e265d30-kube-api-access-4j6gf\" (UniqueName: \"kubernetes.io/projected/169857d1-ca33-4bde-a129-84922e265d30-kube-api-access-4j6gf\") pod \"169857d1-ca33-4bde-a129-84922e265d30\" (UID: \"169857d1-ca33-4bde-a129-84922e265d30\") "
Nov 03 20:00:19.335656 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.335622    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169857d1-ca33-4bde-a129-84922e265d30-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169857d1-ca33-4bde-a129-84922e265d30-maas-api-service-ca\") pod \"169857d1-ca33-4bde-a129-84922e265d30\" (UID: \"169857d1-ca33-4bde-a129-84922e265d30\") "
Nov 03 20:00:19.336497 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.336463    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169857d1-ca33-4bde-a129-84922e265d30-maas-api-service-ca" pod "169857d1-ca33-4bde-a129-84922e265d30" (UID: "169857d1-ca33-4bde-a129-84922e265d30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:19.337008 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.336983    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169857d1-ca33-4bde-a129-84922e265d30-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.338904 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.338877    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169857d1-ca33-4bde-a129-84922e265d30-kube-api-access-4j6gf" pod "169857d1-ca33-4bde-a129-84922e265d30" (UID: "169857d1-ca33-4bde-a129-84922e265d30"). InnerVolumeSpecName "kube-api-access-4j6gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:19.373009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.372980    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"
Nov 03 20:00:19.379301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.379060    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"
Nov 03 20:00:19.406617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.406542    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"]
Nov 03 20:00:19.423560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423522    4531 cpu_manager.go:414] "containerMap: removing container" podUID="10367631-3bd7-476a-88fd-bc1695606723" containerName="cleanup"
Nov 03 20:00:19.423560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423558    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10367631-3bd7-476a-88fd-bc1695606723" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423612    4531 cpu_manager.go:414] "containerMap: removing container" podUID="10367631-3bd7-476a-88fd-bc1695606723" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423621    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10367631-3bd7-476a-88fd-bc1695606723" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423632    4531 cpu_manager.go:414] "containerMap: removing container" podUID="10367631-3bd7-476a-88fd-bc1695606723" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423641    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10367631-3bd7-476a-88fd-bc1695606723" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423656    4531 cpu_manager.go:414] "containerMap: removing container" podUID="169857d1-ca33-4bde-a129-84922e265d30" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423663    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169857d1-ca33-4bde-a129-84922e265d30" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423704    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423711    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423751    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423758    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423785    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerName="cleanup"
Nov 03 20:00:19.423840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.423792    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerName="cleanup"
Nov 03 20:00:19.424573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.424033    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="10367631-3bd7-476a-88fd-bc1695606723" containerName="cleanup"
Nov 03 20:00:19.424573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.424052    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="10367631-3bd7-476a-88fd-bc1695606723" containerName="cleanup"
Nov 03 20:00:19.424573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.424071    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerName="cleanup"
Nov 03 20:00:19.424573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.424086    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerName="cleanup"
Nov 03 20:00:19.424573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.424098    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78" containerName="cleanup"
Nov 03 20:00:19.441687 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.441648    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10367631-3bd7-476a-88fd-bc1695606723-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10367631-3bd7-476a-88fd-bc1695606723-maas-api-service-ca\") pod \"10367631-3bd7-476a-88fd-bc1695606723\" (UID: \"10367631-3bd7-476a-88fd-bc1695606723\") "
Nov 03 20:00:19.441898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.441803    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10367631-3bd7-476a-88fd-bc1695606723-kube-api-access-hd6w5\" (UniqueName: \"kubernetes.io/projected/10367631-3bd7-476a-88fd-bc1695606723-kube-api-access-hd6w5\") pod \"10367631-3bd7-476a-88fd-bc1695606723\" (UID: \"10367631-3bd7-476a-88fd-bc1695606723\") "
Nov 03 20:00:19.442268 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.442245    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c661d33-3c04-4423-8138-aa741b7a0c78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c661d33-3c04-4423-8138-aa741b7a0c78-maas-api-service-ca\") pod \"6c661d33-3c04-4423-8138-aa741b7a0c78\" (UID: \"6c661d33-3c04-4423-8138-aa741b7a0c78\") "
Nov 03 20:00:19.442542 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.442294    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c661d33-3c04-4423-8138-aa741b7a0c78-kube-api-access-hgg9k\" (UniqueName: \"kubernetes.io/projected/6c661d33-3c04-4423-8138-aa741b7a0c78-kube-api-access-hgg9k\") pod \"6c661d33-3c04-4423-8138-aa741b7a0c78\" (UID: \"6c661d33-3c04-4423-8138-aa741b7a0c78\") "
Nov 03 20:00:19.442920 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.442893    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb\" (UID: \"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"
Nov 03 20:00:19.444225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.444195    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mm8b\" (UniqueName: \"kubernetes.io/projected/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-kube-api-access-5mm8b\") pod \"maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb\" (UID: \"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"
Nov 03 20:00:19.444948 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.444440    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j6gf\" (UniqueName: \"kubernetes.io/projected/169857d1-ca33-4bde-a129-84922e265d30-kube-api-access-4j6gf\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.444948 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.444617    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c661d33-3c04-4423-8138-aa741b7a0c78-maas-api-service-ca" pod "6c661d33-3c04-4423-8138-aa741b7a0c78" (UID: "6c661d33-3c04-4423-8138-aa741b7a0c78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:19.445085 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.444987    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10367631-3bd7-476a-88fd-bc1695606723-maas-api-service-ca" pod "10367631-3bd7-476a-88fd-bc1695606723" (UID: "10367631-3bd7-476a-88fd-bc1695606723"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:19.447968 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.447932    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c661d33-3c04-4423-8138-aa741b7a0c78-kube-api-access-hgg9k" pod "6c661d33-3c04-4423-8138-aa741b7a0c78" (UID: "6c661d33-3c04-4423-8138-aa741b7a0c78"). InnerVolumeSpecName "kube-api-access-hgg9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:19.448721 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.448703    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"]
Nov 03 20:00:19.448820 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.448805    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"
Nov 03 20:00:19.450444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.450380    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10367631-3bd7-476a-88fd-bc1695606723-kube-api-access-hd6w5" pod "10367631-3bd7-476a-88fd-bc1695606723" (UID: "10367631-3bd7-476a-88fd-bc1695606723"). InnerVolumeSpecName "kube-api-access-hd6w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:19.453495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.453475    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"]
Nov 03 20:00:19.455072 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.455046    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="10367631-3bd7-476a-88fd-bc1695606723" containerName="cleanup"
Nov 03 20:00:19.460755 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.460352    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"
Nov 03 20:00:19.461447 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.461423    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"]
Nov 03 20:00:19.547274 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.547235    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb\" (UID: \"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"
Nov 03 20:00:19.547572 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.547550    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6823572a-3a8a-410b-9821-3a2d3a06d2fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm\" (UID: \"6823572a-3a8a-410b-9821-3a2d3a06d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"
Nov 03 20:00:19.547677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.547621    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjh7t\" (UniqueName: \"kubernetes.io/projected/6823572a-3a8a-410b-9821-3a2d3a06d2fb-kube-api-access-gjh7t\") pod \"maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm\" (UID: \"6823572a-3a8a-410b-9821-3a2d3a06d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"
Nov 03 20:00:19.547789 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.547761    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mm8b\" (UniqueName: \"kubernetes.io/projected/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-kube-api-access-5mm8b\") pod \"maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb\" (UID: \"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"
Nov 03 20:00:19.547874 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.547854    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t\" (UID: \"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"
Nov 03 20:00:19.547939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.547909    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p247\" (UniqueName: \"kubernetes.io/projected/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-kube-api-access-8p247\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t\" (UID: \"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"
Nov 03 20:00:19.548092 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.548068    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb\" (UID: \"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"
Nov 03 20:00:19.548304 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.548287    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd6w5\" (UniqueName: \"kubernetes.io/projected/10367631-3bd7-476a-88fd-bc1695606723-kube-api-access-hd6w5\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.548398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.548314    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c661d33-3c04-4423-8138-aa741b7a0c78-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.548398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.548335    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgg9k\" (UniqueName: \"kubernetes.io/projected/6c661d33-3c04-4423-8138-aa741b7a0c78-kube-api-access-hgg9k\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.548398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.548345    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10367631-3bd7-476a-88fd-bc1695606723-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.561373 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.561340    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mm8b\" (UniqueName: \"kubernetes.io/projected/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-kube-api-access-5mm8b\") pod \"maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb\" (UID: \"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"
Nov 03 20:00:19.596992 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.596957    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"
Nov 03 20:00:19.649873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.649840    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6823572a-3a8a-410b-9821-3a2d3a06d2fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm\" (UID: \"6823572a-3a8a-410b-9821-3a2d3a06d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"
Nov 03 20:00:19.650054 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.649910    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjh7t\" (UniqueName: \"kubernetes.io/projected/6823572a-3a8a-410b-9821-3a2d3a06d2fb-kube-api-access-gjh7t\") pod \"maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm\" (UID: \"6823572a-3a8a-410b-9821-3a2d3a06d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"
Nov 03 20:00:19.650054 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.650029    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t\" (UID: \"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"
Nov 03 20:00:19.650226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.650086    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p247\" (UniqueName: \"kubernetes.io/projected/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-kube-api-access-8p247\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t\" (UID: \"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"
Nov 03 20:00:19.650748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.650706    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6823572a-3a8a-410b-9821-3a2d3a06d2fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm\" (UID: \"6823572a-3a8a-410b-9821-3a2d3a06d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"
Nov 03 20:00:19.650889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.650727    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t\" (UID: \"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"
Nov 03 20:00:19.658053 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.657973    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p247\" (UniqueName: \"kubernetes.io/projected/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-kube-api-access-8p247\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t\" (UID: \"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"
Nov 03 20:00:19.658053 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.658013    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjh7t\" (UniqueName: \"kubernetes.io/projected/6823572a-3a8a-410b-9821-3a2d3a06d2fb-kube-api-access-gjh7t\") pod \"maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm\" (UID: \"6823572a-3a8a-410b-9821-3a2d3a06d2fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"
Nov 03 20:00:19.774998 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.774963    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"
Nov 03 20:00:19.784781 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.784747    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"
Nov 03 20:00:19.825390 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.825360    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"
Nov 03 20:00:19.826377 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.826348    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb" event={"ID":"169857d1-ca33-4bde-a129-84922e265d30","Type":"ContainerDied","Data":"75397edf05d3e0d664f9ac8e9ba47ca5e2d197107af59df1cdf3512494673ee3"}
Nov 03 20:00:19.827016 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.826916    4531 scope.go:122] "RemoveContainer" containerID="7c9a7a93a26d07a1348a91d305f3de2fe881a2a27345e306404acbcad14f983f"
Nov 03 20:00:19.836749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.836387    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2" event={"ID":"6c661d33-3c04-4423-8138-aa741b7a0c78","Type":"ContainerDied","Data":"1aeb043cbf2c72f46efb3140752578019f1e10b84211e9efea063d086988ccfd"}
Nov 03 20:00:19.836749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.836460    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"
Nov 03 20:00:19.840671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.840643    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj" event={"ID":"10367631-3bd7-476a-88fd-bc1695606723","Type":"ContainerDied","Data":"fab54b704770ee9b6d01bc4deff34e92f541827675e33a33733427fe2451fb3a"}
Nov 03 20:00:19.840782 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.840753    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"
Nov 03 20:00:19.858927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.857739    4531 scope.go:122] "RemoveContainer" containerID="47c5ed5af51545fbdea281aee68266a354c0cfbd4521d47b505043cc52195a29"
Nov 03 20:00:19.864373 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.864349    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"]
Nov 03 20:00:19.884565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.884540    4531 scope.go:122] "RemoveContainer" containerID="242c2fe5fe2e1fd820a3653eb421b16219b05976e34521b52207357e76cf186c"
Nov 03 20:00:19.885671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.885588    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fa9526-29895600-2fdcb"]
Nov 03 20:00:19.928609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.919249    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"]
Nov 03 20:00:19.928609 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:00:19.927802    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7082dcc4_b73c_43f9_8f11_ff7c18ecb58c.slice/crio-c790a270246b7f5eee379939e4e9e45e8a45c63eb813a04a276848b57bd1adbf WatchSource:0}: Error finding container c790a270246b7f5eee379939e4e9e45e8a45c63eb813a04a276848b57bd1adbf: Status 404 returned error can't find the container with id c790a270246b7f5eee379939e4e9e45e8a45c63eb813a04a276848b57bd1adbf
Nov 03 20:00:19.933890 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.933841    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"]
Nov 03 20:00:19.961987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.961903    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b9cbea-29895600-bk9qj"]
Nov 03 20:00:19.968883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.968120    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"]
Nov 03 20:00:19.976403 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:19.976371    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-6eb768-29895600-s4pw2"]
Nov 03 20:00:20.106304 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:20.106253    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:20.156005 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:20.155940    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"]
Nov 03 20:00:20.198687 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:20.198658    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"]
Nov 03 20:00:20.365335 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:20.365306    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"
Nov 03 20:00:20.495885 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:20.495793    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc39eb6-3905-4203-b242-a28f6b12477a-kube-api-access-smm5f\" (UniqueName: \"kubernetes.io/projected/ebc39eb6-3905-4203-b242-a28f6b12477a-kube-api-access-smm5f\") pod \"ebc39eb6-3905-4203-b242-a28f6b12477a\" (UID: \"ebc39eb6-3905-4203-b242-a28f6b12477a\") "
Nov 03 20:00:20.496054 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:20.495989    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc39eb6-3905-4203-b242-a28f6b12477a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc39eb6-3905-4203-b242-a28f6b12477a-maas-api-service-ca\") pod \"ebc39eb6-3905-4203-b242-a28f6b12477a\" (UID: \"ebc39eb6-3905-4203-b242-a28f6b12477a\") "
Nov 03 20:00:20.496427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:20.496395    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc39eb6-3905-4203-b242-a28f6b12477a-maas-api-service-ca" pod "ebc39eb6-3905-4203-b242-a28f6b12477a" (UID: "ebc39eb6-3905-4203-b242-a28f6b12477a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:20.496890 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:20.496868    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc39eb6-3905-4203-b242-a28f6b12477a-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:20.539343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:20.539307    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc39eb6-3905-4203-b242-a28f6b12477a-kube-api-access-smm5f" pod "ebc39eb6-3905-4203-b242-a28f6b12477a" (UID: "ebc39eb6-3905-4203-b242-a28f6b12477a"). InnerVolumeSpecName "kube-api-access-smm5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:20.577708 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:20.577672    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:00:20.598403 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:20.598363    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smm5f\" (UniqueName: \"kubernetes.io/projected/ebc39eb6-3905-4203-b242-a28f6b12477a-kube-api-access-smm5f\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:20.839864 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:20.839827    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:21.011086 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.011037    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"
Nov 03 20:00:21.015006 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.014955    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10367631-3bd7-476a-88fd-bc1695606723" path="/var/lib/kubelet/pods/10367631-3bd7-476a-88fd-bc1695606723/volumes"
Nov 03 20:00:21.016878 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.016847    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169857d1-ca33-4bde-a129-84922e265d30" path="/var/lib/kubelet/pods/169857d1-ca33-4bde-a129-84922e265d30/volumes"
Nov 03 20:00:21.017830 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.017804    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c661d33-3c04-4423-8138-aa741b7a0c78" path="/var/lib/kubelet/pods/6c661d33-3c04-4423-8138-aa741b7a0c78/volumes"
Nov 03 20:00:21.018795 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.018768    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t" event={"ID":"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7","Type":"ContainerStarted","Data":"60ef66cc8f44397220b5b5b9fac9de4baa98811d46a2b4310bce0355126b348e"}
Nov 03 20:00:21.018909 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.018817    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb" event={"ID":"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c","Type":"ContainerStarted","Data":"210a1f37c26e46b714d434bccca8d750fb1d5e79003012a6eaa6a219cfd020da"}
Nov 03 20:00:21.018909 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.018853    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb" event={"ID":"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c","Type":"ContainerStarted","Data":"c790a270246b7f5eee379939e4e9e45e8a45c63eb813a04a276848b57bd1adbf"}
Nov 03 20:00:21.018909 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.018875    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm" event={"ID":"6823572a-3a8a-410b-9821-3a2d3a06d2fb","Type":"ContainerStarted","Data":"19a0970e02106a981def194a772caf9776ea8da88fc2d298d72bc8761bf64191"}
Nov 03 20:00:21.018909 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.018901    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp" event={"ID":"ebc39eb6-3905-4203-b242-a28f6b12477a","Type":"ContainerDied","Data":"3c8d3329d32837057b6fd16c942c8c7dba0748946940620c3b5981cd9c18497b"}
Nov 03 20:00:21.019136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.018936    4531 scope.go:122] "RemoveContainer" containerID="d6c3c65cec9e41cc144956c2ca107a9647cbb335a27ba27fbc0d8be6eb89d5b1"
Nov 03 20:00:21.031679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.031372    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb" podStartSLOduration=21.03135377 podStartE2EDuration="21.03135377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:21.025101676 +0000 UTC m=+1734.295224972" watchObservedRunningTime="2026-11-03 20:00:21.03135377 +0000 UTC m=+1734.301477066"
Nov 03 20:00:21.050408 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.050360    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm" podStartSLOduration=21.050345838 podStartE2EDuration="21.050345838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:21.045282878 +0000 UTC m=+1734.315406168" watchObservedRunningTime="2026-11-03 20:00:21.050345838 +0000 UTC m=+1734.320469132"
Nov 03 20:00:21.063595 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.063558    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"]
Nov 03 20:00:21.069535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.069506    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2d78ea-29895600-7zhcp"]
Nov 03 20:00:21.614310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:21.614273    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:22.023573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:22.023539    4531 generic.go:356] "Generic (PLEG): container finished" podID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerID="210a1f37c26e46b714d434bccca8d750fb1d5e79003012a6eaa6a219cfd020da" exitCode=60
Nov 03 20:00:22.024120 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:22.023658    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb" event={"ID":"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c","Type":"ContainerDied","Data":"210a1f37c26e46b714d434bccca8d750fb1d5e79003012a6eaa6a219cfd020da"}
Nov 03 20:00:22.024120 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:22.023993    4531 scope.go:122] "RemoveContainer" containerID="210a1f37c26e46b714d434bccca8d750fb1d5e79003012a6eaa6a219cfd020da"
Nov 03 20:00:22.026274 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:22.026033    4531 generic.go:356] "Generic (PLEG): container finished" podID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerID="c38423d61e651dff4d0ddc64457304f1a6357aea74f0bd70fa439ff61d1d42a6" exitCode=60
Nov 03 20:00:22.026274 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:22.026262    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm" event={"ID":"6823572a-3a8a-410b-9821-3a2d3a06d2fb","Type":"ContainerDied","Data":"c38423d61e651dff4d0ddc64457304f1a6357aea74f0bd70fa439ff61d1d42a6"}
Nov 03 20:00:22.027518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:22.027499    4531 scope.go:122] "RemoveContainer" containerID="c38423d61e651dff4d0ddc64457304f1a6357aea74f0bd70fa439ff61d1d42a6"
Nov 03 20:00:22.031121 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:22.031097    4531 generic.go:356] "Generic (PLEG): container finished" podID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerID="57f8dd4588d84a2a7f90d08af50876ca1c791c06a1ae3feb2bf0bf4be4af7d35" exitCode=60
Nov 03 20:00:22.031230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:22.031206    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t" event={"ID":"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7","Type":"ContainerDied","Data":"57f8dd4588d84a2a7f90d08af50876ca1c791c06a1ae3feb2bf0bf4be4af7d35"}
Nov 03 20:00:22.031602 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:22.031499    4531 scope.go:122] "RemoveContainer" containerID="57f8dd4588d84a2a7f90d08af50876ca1c791c06a1ae3feb2bf0bf4be4af7d35"
Nov 03 20:00:22.509587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:22.509549    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:22.864984 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:22.864893    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a" path="/var/lib/kubelet/pods/ebc39eb6-3905-4203-b242-a28f6b12477a/volumes"
Nov 03 20:00:23.043346 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:23.043309    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb" event={"ID":"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c","Type":"ContainerStarted","Data":"1f0762b4787fed3efa422cbbb6881d86192e0fabc4222eebf65932408f28fde4"}
Nov 03 20:00:23.045662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:23.045636    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm" event={"ID":"6823572a-3a8a-410b-9821-3a2d3a06d2fb","Type":"ContainerStarted","Data":"745f0da4b8f51414e6f20b9df66c80d4fb1511dab19d461c766cf72cdeddbb1a"}
Nov 03 20:00:23.047777 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:23.047750    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t" event={"ID":"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7","Type":"ContainerStarted","Data":"c510aebf4d773858af454fa16ae57e559827274cd6ad4001ddc08f55e06c9c33"}
Nov 03 20:00:23.095971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:23.095905    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t" podStartSLOduration=23.095885803 podStartE2EDuration="23.095885803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:23.095218943 +0000 UTC m=+1736.365342228" watchObservedRunningTime="2026-11-03 20:00:23.095885803 +0000 UTC m=+1736.366009100"
Nov 03 20:00:23.160787 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:23.160744    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:24.018681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.014030    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:24.070869 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.070827    4531 generic.go:356] "Generic (PLEG): container finished" podID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerID="c510aebf4d773858af454fa16ae57e559827274cd6ad4001ddc08f55e06c9c33" exitCode=60
Nov 03 20:00:24.071409 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.071318    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t" event={"ID":"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7","Type":"ContainerDied","Data":"c510aebf4d773858af454fa16ae57e559827274cd6ad4001ddc08f55e06c9c33"}
Nov 03 20:00:24.071409 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.071376    4531 scope.go:122] "RemoveContainer" containerID="57f8dd4588d84a2a7f90d08af50876ca1c791c06a1ae3feb2bf0bf4be4af7d35"
Nov 03 20:00:24.073311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.073285    4531 scope.go:122] "RemoveContainer" containerID="c510aebf4d773858af454fa16ae57e559827274cd6ad4001ddc08f55e06c9c33"
Nov 03 20:00:24.073798 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:24.073773    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t_odh-ai-gateway-infra(cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7"
Nov 03 20:00:24.073977 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.073953    4531 generic.go:356] "Generic (PLEG): container finished" podID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerID="1f0762b4787fed3efa422cbbb6881d86192e0fabc4222eebf65932408f28fde4" exitCode=60
Nov 03 20:00:24.074115 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.074089    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb" event={"ID":"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c","Type":"ContainerDied","Data":"1f0762b4787fed3efa422cbbb6881d86192e0fabc4222eebf65932408f28fde4"}
Nov 03 20:00:24.074483 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.074465    4531 scope.go:122] "RemoveContainer" containerID="1f0762b4787fed3efa422cbbb6881d86192e0fabc4222eebf65932408f28fde4"
Nov 03 20:00:24.074952 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:24.074931    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb_odh-ai-gateway-infra(7082dcc4-b73c-43f9-8f11-ff7c18ecb58c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c"
Nov 03 20:00:24.077488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.077461    4531 generic.go:356] "Generic (PLEG): container finished" podID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerID="745f0da4b8f51414e6f20b9df66c80d4fb1511dab19d461c766cf72cdeddbb1a" exitCode=60
Nov 03 20:00:24.077623 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.077602    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm" event={"ID":"6823572a-3a8a-410b-9821-3a2d3a06d2fb","Type":"ContainerDied","Data":"745f0da4b8f51414e6f20b9df66c80d4fb1511dab19d461c766cf72cdeddbb1a"}
Nov 03 20:00:24.077885 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.077868    4531 scope.go:122] "RemoveContainer" containerID="745f0da4b8f51414e6f20b9df66c80d4fb1511dab19d461c766cf72cdeddbb1a"
Nov 03 20:00:24.078154 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:24.078136    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm_odh-ai-gateway-infra(6823572a-3a8a-410b-9821-3a2d3a06d2fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb"
Nov 03 20:00:24.088301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.088267    4531 scope.go:122] "RemoveContainer" containerID="210a1f37c26e46b714d434bccca8d750fb1d5e79003012a6eaa6a219cfd020da"
Nov 03 20:00:24.116348 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.116315    4531 scope.go:122] "RemoveContainer" containerID="c38423d61e651dff4d0ddc64457304f1a6357aea74f0bd70fa439ff61d1d42a6"
Nov 03 20:00:24.776257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:24.765078    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:25.088785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:25.088687    4531 scope.go:122] "RemoveContainer" containerID="1f0762b4787fed3efa422cbbb6881d86192e0fabc4222eebf65932408f28fde4"
Nov 03 20:00:25.089227 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:25.089198    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb_odh-ai-gateway-infra(7082dcc4-b73c-43f9-8f11-ff7c18ecb58c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c"
Nov 03 20:00:25.090587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:25.090561    4531 scope.go:122] "RemoveContainer" containerID="745f0da4b8f51414e6f20b9df66c80d4fb1511dab19d461c766cf72cdeddbb1a"
Nov 03 20:00:25.091080 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:25.091049    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm_odh-ai-gateway-infra(6823572a-3a8a-410b-9821-3a2d3a06d2fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb"
Nov 03 20:00:25.092461 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:25.092436    4531 scope.go:122] "RemoveContainer" containerID="c510aebf4d773858af454fa16ae57e559827274cd6ad4001ddc08f55e06c9c33"
Nov 03 20:00:25.092925 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:25.092901    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t_odh-ai-gateway-infra(cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7"
Nov 03 20:00:25.414254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:25.414215    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:26.171200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:26.169297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:26.304153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:26.304116    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:26.831061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:26.831005    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:27.576957 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:27.576921    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:28.176510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:28.176467    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:28.879107 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:28.879063    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:29.520283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:29.520243    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:30.134231 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:30.134195    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:30.833016 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:30.832978    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:31.507929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:31.507891    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:32.175342 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:32.175303    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:32.915206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:32.911915    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:33.039505 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:33.039471    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:33.569192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:33.566901    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:34.113523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:34.113482    4531 scope.go:122] "RemoveContainer" containerID="0a6c31d1d5bb15e41d4c15e78362f07e8aa722fda94e0cc51db8e1a82aed8a4d"
Nov 03 20:00:34.243952 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:34.243910    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:34.851583 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:34.851551    4531 scope.go:122] "RemoveContainer" containerID="745f0da4b8f51414e6f20b9df66c80d4fb1511dab19d461c766cf72cdeddbb1a"
Nov 03 20:00:34.851950 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:34.851928    4531 scope.go:122] "RemoveContainer" containerID="1f0762b4787fed3efa422cbbb6881d86192e0fabc4222eebf65932408f28fde4"
Nov 03 20:00:35.114392 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:35.113840    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:35.718690 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:35.718605    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:36.304608 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:36.304509    4531 generic.go:356] "Generic (PLEG): container finished" podID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerID="baeb8d777d3c69c641f17ddff8381d6ced658df6db0b5327ef558c23964767b1" exitCode=60
Nov 03 20:00:36.304608 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:36.304576    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb" event={"ID":"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c","Type":"ContainerDied","Data":"baeb8d777d3c69c641f17ddff8381d6ced658df6db0b5327ef558c23964767b1"}
Nov 03 20:00:36.305208 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:36.304641    4531 scope.go:122] "RemoveContainer" containerID="1f0762b4787fed3efa422cbbb6881d86192e0fabc4222eebf65932408f28fde4"
Nov 03 20:00:36.305208 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:36.305191    4531 scope.go:122] "RemoveContainer" containerID="baeb8d777d3c69c641f17ddff8381d6ced658df6db0b5327ef558c23964767b1"
Nov 03 20:00:36.305740 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:36.305719    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb_odh-ai-gateway-infra(7082dcc4-b73c-43f9-8f11-ff7c18ecb58c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c"
Nov 03 20:00:36.307138 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:36.307114    4531 generic.go:356] "Generic (PLEG): container finished" podID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerID="a2512c604da0b42e039f3315d233cc2c52289bd9537722316845c269212ebc63" exitCode=60
Nov 03 20:00:36.307302 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:36.307224    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm" event={"ID":"6823572a-3a8a-410b-9821-3a2d3a06d2fb","Type":"ContainerDied","Data":"a2512c604da0b42e039f3315d233cc2c52289bd9537722316845c269212ebc63"}
Nov 03 20:00:36.307685 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:36.307668    4531 scope.go:122] "RemoveContainer" containerID="a2512c604da0b42e039f3315d233cc2c52289bd9537722316845c269212ebc63"
Nov 03 20:00:36.308120 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:36.308100    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm_odh-ai-gateway-infra(6823572a-3a8a-410b-9821-3a2d3a06d2fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb"
Nov 03 20:00:36.319986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:36.319963    4531 scope.go:122] "RemoveContainer" containerID="745f0da4b8f51414e6f20b9df66c80d4fb1511dab19d461c766cf72cdeddbb1a"
Nov 03 20:00:36.531164 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:36.531127    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:36.865566 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:36.865533    4531 scope.go:122] "RemoveContainer" containerID="c510aebf4d773858af454fa16ae57e559827274cd6ad4001ddc08f55e06c9c33"
Nov 03 20:00:37.260197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.259901    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:37.338784 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.338735    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"]
Nov 03 20:00:37.357697 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.357663    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"]
Nov 03 20:00:37.404642 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.404607    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"]
Nov 03 20:00:37.406593 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.406540    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerName="cleanup"
Nov 03 20:00:37.406778 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.406762    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerName="cleanup"
Nov 03 20:00:37.406889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.406879    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerName="cleanup"
Nov 03 20:00:37.406975 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.406964    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerName="cleanup"
Nov 03 20:00:37.408319 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.408293    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerName="cleanup"
Nov 03 20:00:37.408506 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.408473    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerName="cleanup"
Nov 03 20:00:37.408628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.408616    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerName="cleanup"
Nov 03 20:00:37.414980 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.414914    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:37.425708 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.424292    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:00:37.428660 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.428621    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"]
Nov 03 20:00:37.490828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.490753    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2k2d\" (UniqueName: \"kubernetes.io/projected/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-kube-api-access-k2k2d\") pod \"maas-api-revoke-sub-06962bc6-kb5rx\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:37.491051 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.490881    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06962bc6-kb5rx\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:37.491051 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.491011    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06962bc6-kb5rx\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:37.593832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.593791    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06962bc6-kb5rx\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:37.594030 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.593947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06962bc6-kb5rx\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:37.594456 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.594422    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2k2d\" (UniqueName: \"kubernetes.io/projected/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-kube-api-access-k2k2d\") pod \"maas-api-revoke-sub-06962bc6-kb5rx\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:37.596281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.596248    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06962bc6-kb5rx\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:37.599088 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.599061    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06962bc6-kb5rx\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:37.625220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.605562    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2k2d\" (UniqueName: \"kubernetes.io/projected/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-kube-api-access-k2k2d\") pod \"maas-api-revoke-sub-06962bc6-kb5rx\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:37.698554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.698526    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"
Nov 03 20:00:37.702305 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.702280    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"
Nov 03 20:00:37.756488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.756455    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:37.803280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.803243    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6823572a-3a8a-410b-9821-3a2d3a06d2fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6823572a-3a8a-410b-9821-3a2d3a06d2fb-maas-api-service-ca\") pod \"6823572a-3a8a-410b-9821-3a2d3a06d2fb\" (UID: \"6823572a-3a8a-410b-9821-3a2d3a06d2fb\") "
Nov 03 20:00:37.803467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.803412    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-maas-api-service-ca\") pod \"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c\" (UID: \"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c\") "
Nov 03 20:00:37.803590 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.803502    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-kube-api-access-5mm8b\" (UniqueName: \"kubernetes.io/projected/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-kube-api-access-5mm8b\") pod \"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c\" (UID: \"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c\") "
Nov 03 20:00:37.803590 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.803538    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6823572a-3a8a-410b-9821-3a2d3a06d2fb-kube-api-access-gjh7t\" (UniqueName: \"kubernetes.io/projected/6823572a-3a8a-410b-9821-3a2d3a06d2fb-kube-api-access-gjh7t\") pod \"6823572a-3a8a-410b-9821-3a2d3a06d2fb\" (UID: \"6823572a-3a8a-410b-9821-3a2d3a06d2fb\") "
Nov 03 20:00:37.805329 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.805287    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-maas-api-service-ca" pod "7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" (UID: "7082dcc4-b73c-43f9-8f11-ff7c18ecb58c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:37.805670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.805645    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6823572a-3a8a-410b-9821-3a2d3a06d2fb-maas-api-service-ca" pod "6823572a-3a8a-410b-9821-3a2d3a06d2fb" (UID: "6823572a-3a8a-410b-9821-3a2d3a06d2fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:37.808850 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.808823    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6823572a-3a8a-410b-9821-3a2d3a06d2fb-kube-api-access-gjh7t" pod "6823572a-3a8a-410b-9821-3a2d3a06d2fb" (UID: "6823572a-3a8a-410b-9821-3a2d3a06d2fb"). InnerVolumeSpecName "kube-api-access-gjh7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:37.808998 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.808979    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-kube-api-access-5mm8b" pod "7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" (UID: "7082dcc4-b73c-43f9-8f11-ff7c18ecb58c"). InnerVolumeSpecName "kube-api-access-5mm8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:37.907484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.907436    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6823572a-3a8a-410b-9821-3a2d3a06d2fb-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:37.907484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.907479    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:37.907765 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.907494    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mm8b\" (UniqueName: \"kubernetes.io/projected/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c-kube-api-access-5mm8b\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:37.907765 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.907513    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjh7t\" (UniqueName: \"kubernetes.io/projected/6823572a-3a8a-410b-9821-3a2d3a06d2fb-kube-api-access-gjh7t\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:37.973287 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:37.970753    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:38.092412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.091129    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"]
Nov 03 20:00:38.092412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.092393    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5819b09-4rtsm"]
Nov 03 20:00:38.111756 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.105898    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"]
Nov 03 20:00:38.349507 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.349412    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"
Nov 03 20:00:38.349507 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.349490    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb" event={"ID":"7082dcc4-b73c-43f9-8f11-ff7c18ecb58c","Type":"ContainerDied","Data":"c790a270246b7f5eee379939e4e9e45e8a45c63eb813a04a276848b57bd1adbf"}
Nov 03 20:00:38.351616 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.349543    4531 scope.go:122] "RemoveContainer" containerID="baeb8d777d3c69c641f17ddff8381d6ced658df6db0b5327ef558c23964767b1"
Nov 03 20:00:38.361072 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.361032    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm" event={"ID":"6823572a-3a8a-410b-9821-3a2d3a06d2fb","Type":"ContainerDied","Data":"19a0970e02106a981def194a772caf9776ea8da88fc2d298d72bc8761bf64191"}
Nov 03 20:00:38.361257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.361149    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"
Nov 03 20:00:38.367660 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.367606    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx" event={"ID":"acc5d5e9-df4c-4644-b1a3-1577b98c9a80","Type":"ContainerStarted","Data":"c8d79b5c337b6ed1c20759b452af513d621a4fb15700d7941fa6ca49f5f1c6ab"}
Nov 03 20:00:38.372283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.372252    4531 generic.go:356] "Generic (PLEG): container finished" podID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerID="a56a9be78410423d47b4f9c6212cf42be9b4b9b05af61161cf14aa0147d39ad2" exitCode=60
Nov 03 20:00:38.372421 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.372336    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t" event={"ID":"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7","Type":"ContainerDied","Data":"a56a9be78410423d47b4f9c6212cf42be9b4b9b05af61161cf14aa0147d39ad2"}
Nov 03 20:00:38.372892 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.372871    4531 scope.go:122] "RemoveContainer" containerID="a56a9be78410423d47b4f9c6212cf42be9b4b9b05af61161cf14aa0147d39ad2"
Nov 03 20:00:38.373410 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:38.373389    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t_odh-ai-gateway-infra(cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7"
Nov 03 20:00:38.380284 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.380261    4531 scope.go:122] "RemoveContainer" containerID="a2512c604da0b42e039f3315d233cc2c52289bd9537722316845c269212ebc63"
Nov 03 20:00:38.396745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.396709    4531 scope.go:122] "RemoveContainer" containerID="c510aebf4d773858af454fa16ae57e559827274cd6ad4001ddc08f55e06c9c33"
Nov 03 20:00:38.396900 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.396852    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"]
Nov 03 20:00:38.410816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.410223    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c18bea-29895600-c49hb"]
Nov 03 20:00:38.418905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.418877    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"]
Nov 03 20:00:38.427989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.427959    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bdb694-29895600-hs8dm"]
Nov 03 20:00:38.951199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.950151    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:38.996222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.994056    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" path="/var/lib/kubelet/pods/6823572a-3a8a-410b-9821-3a2d3a06d2fb/volumes"
Nov 03 20:00:38.996222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.995069    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" path="/var/lib/kubelet/pods/7082dcc4-b73c-43f9-8f11-ff7c18ecb58c/volumes"
Nov 03 20:00:38.996222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:38.995970    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c" path="/var/lib/kubelet/pods/fa0e8797-c7b1-4e3c-aafa-ffdb6d5f920c/volumes"
Nov 03 20:00:39.388004 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:39.387965    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx" event={"ID":"acc5d5e9-df4c-4644-b1a3-1577b98c9a80","Type":"ContainerStarted","Data":"62e35dc414d3af4d379a31079527879cdd210b7d3df21d2e10e09adf3f3ab0d8"}
Nov 03 20:00:39.410339 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:39.410277    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx" podStartSLOduration=2.410259032 podStartE2EDuration="2.410259032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:39.405536535 +0000 UTC m=+1752.675659830" watchObservedRunningTime="2026-11-03 20:00:39.410259032 +0000 UTC m=+1752.680382328"
Nov 03 20:00:39.441798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:39.441768    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"]
Nov 03 20:00:39.691662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:39.691632    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"
Nov 03 20:00:39.740759 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:39.740605    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:39.789194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:39.783019    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-kube-api-access-8p247\" (UniqueName: \"kubernetes.io/projected/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-kube-api-access-8p247\") pod \"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7\" (UID: \"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7\") "
Nov 03 20:00:39.789194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:39.785972    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-maas-api-service-ca\") pod \"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7\" (UID: \"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7\") "
Nov 03 20:00:39.789194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:39.786610    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-maas-api-service-ca" pod "cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" (UID: "cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:39.811196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:39.810916    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:39.822520 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:39.818401    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-kube-api-access-8p247" pod "cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" (UID: "cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7"). InnerVolumeSpecName "kube-api-access-8p247". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:39.913635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:39.913548    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p247\" (UniqueName: \"kubernetes.io/projected/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7-kube-api-access-8p247\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:40.373235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:40.372922    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:40.468473 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:40.468437    4531 generic.go:356] "Generic (PLEG): container finished" podID="acc5d5e9-df4c-4644-b1a3-1577b98c9a80" containerID="62e35dc414d3af4d379a31079527879cdd210b7d3df21d2e10e09adf3f3ab0d8" exitCode=0
Nov 03 20:00:40.468906 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:40.468525    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx" event={"ID":"acc5d5e9-df4c-4644-b1a3-1577b98c9a80","Type":"ContainerDied","Data":"62e35dc414d3af4d379a31079527879cdd210b7d3df21d2e10e09adf3f3ab0d8"}
Nov 03 20:00:40.470150 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:40.470133    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"
Nov 03 20:00:40.470245 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:40.470159    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t" event={"ID":"cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7","Type":"ContainerDied","Data":"60ef66cc8f44397220b5b5b9fac9de4baa98811d46a2b4310bce0355126b348e"}
Nov 03 20:00:40.470285 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:40.470269    4531 scope.go:122] "RemoveContainer" containerID="a56a9be78410423d47b4f9c6212cf42be9b4b9b05af61161cf14aa0147d39ad2"
Nov 03 20:00:40.503363 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:40.503323    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"]
Nov 03 20:00:40.509505 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:40.509473    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-32a584-29895600-m6c8t"]
Nov 03 20:00:40.862620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:40.862582    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" path="/var/lib/kubelet/pods/cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7/volumes"
Nov 03 20:00:41.151362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.151332    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:41.682130 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.682104    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:41.765004 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.764913    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-kube-api-access-k2k2d\" (UniqueName: \"kubernetes.io/projected/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-kube-api-access-k2k2d\") pod \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") "
Nov 03 20:00:41.765269 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.765243    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-xks-ca\") pod \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") "
Nov 03 20:00:41.765388 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.765333    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-openshift-ca\") pod \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\" (UID: \"acc5d5e9-df4c-4644-b1a3-1577b98c9a80\") "
Nov 03 20:00:41.765719 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.765690    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-openshift-ca" pod "acc5d5e9-df4c-4644-b1a3-1577b98c9a80" (UID: "acc5d5e9-df4c-4644-b1a3-1577b98c9a80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:41.766645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.766621    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:41.767640 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.767611    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-xks-ca" pod "acc5d5e9-df4c-4644-b1a3-1577b98c9a80" (UID: "acc5d5e9-df4c-4644-b1a3-1577b98c9a80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:41.767919 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.767895    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-kube-api-access-k2k2d" pod "acc5d5e9-df4c-4644-b1a3-1577b98c9a80" (UID: "acc5d5e9-df4c-4644-b1a3-1577b98c9a80"). InnerVolumeSpecName "kube-api-access-k2k2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:41.869189 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.869138    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:41.869189 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.869187    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2k2d\" (UniqueName: \"kubernetes.io/projected/acc5d5e9-df4c-4644-b1a3-1577b98c9a80-kube-api-access-k2k2d\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:41.970863 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:41.969303    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:42.492671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:42.492645    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"
Nov 03 20:00:42.492860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:42.492674    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx" event={"ID":"acc5d5e9-df4c-4644-b1a3-1577b98c9a80","Type":"ContainerDied","Data":"c8d79b5c337b6ed1c20759b452af513d621a4fb15700d7941fa6ca49f5f1c6ab"}
Nov 03 20:00:42.492860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:42.492725    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8d79b5c337b6ed1c20759b452af513d621a4fb15700d7941fa6ca49f5f1c6ab"
Nov 03 20:00:42.837947 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:42.837853    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:43.591929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:43.591890    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:44.639022 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:44.638880    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:45.427196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:45.425336    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:46.072226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:46.070949    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:46.796084 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:46.796049    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:47.443622 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:47.443567    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:47.934997 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:47.934955    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"]
Nov 03 20:00:48.007888 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:48.007843    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:48.367158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:48.367067    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:49.120111 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:49.120069    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:50.161824 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:50.161788    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:50.577767 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:00:50.577668    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:00:50.843256 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:50.830984    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:51.737687 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:51.737647    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:52.549474 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:52.549424    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:53.207738 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:53.207700    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:53.988647 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:53.988611    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:54.781787 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:54.781745    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:55.562167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:55.562087    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:56.439230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:56.437960    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:57.111183 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:57.111147    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:57.963872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:57.963807    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:58.590875 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:58.590840    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:59.073639 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:59.073397    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"]
Nov 03 20:00:59.150076 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:59.150032    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:59.286936 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:00:59.286784    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:00.410222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:00.410189    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:01.222923 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:01.222884    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:02.030332 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:02.030294    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:02.904202 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:02.892696    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:02.969442 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:02.969400    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerName="main" containerID="cri-o://7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691" gracePeriod=45
Nov 03 20:01:03.284752 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.284724    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 20:01:03.364866 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.364818    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-model-cache\") pod \"abf4257b-2d15-421e-96c0-9650d6f82a57\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") "
Nov 03 20:01:03.365181 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.365020    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-tmp-dir\") pod \"abf4257b-2d15-421e-96c0-9650d6f82a57\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") "
Nov 03 20:01:03.365181 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.365083    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-dshm\") pod \"abf4257b-2d15-421e-96c0-9650d6f82a57\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") "
Nov 03 20:01:03.365323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.365207    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-home\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-home\") pod \"abf4257b-2d15-421e-96c0-9650d6f82a57\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") "
Nov 03 20:01:03.365323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.365165    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-model-cache" pod "abf4257b-2d15-421e-96c0-9650d6f82a57" (UID: "abf4257b-2d15-421e-96c0-9650d6f82a57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:03.365323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.365258    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf4257b-2d15-421e-96c0-9650d6f82a57-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf4257b-2d15-421e-96c0-9650d6f82a57-tls-certs\") pod \"abf4257b-2d15-421e-96c0-9650d6f82a57\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") "
Nov 03 20:01:03.365504 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.365352    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-tmp-dir" pod "abf4257b-2d15-421e-96c0-9650d6f82a57" (UID: "abf4257b-2d15-421e-96c0-9650d6f82a57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:03.365504 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.365459    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf4257b-2d15-421e-96c0-9650d6f82a57-kube-api-access-nt6t7\" (UniqueName: \"kubernetes.io/projected/abf4257b-2d15-421e-96c0-9650d6f82a57-kube-api-access-nt6t7\") pod \"abf4257b-2d15-421e-96c0-9650d6f82a57\" (UID: \"abf4257b-2d15-421e-96c0-9650d6f82a57\") "
Nov 03 20:01:03.365616 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.365574    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-home" pod "abf4257b-2d15-421e-96c0-9650d6f82a57" (UID: "abf4257b-2d15-421e-96c0-9650d6f82a57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:03.366491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.366461    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.366612 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.366495    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.366612 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.366510    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.368418 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.368387    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-dshm" pod "abf4257b-2d15-421e-96c0-9650d6f82a57" (UID: "abf4257b-2d15-421e-96c0-9650d6f82a57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:03.368742 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.368721    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf4257b-2d15-421e-96c0-9650d6f82a57-tls-certs" pod "abf4257b-2d15-421e-96c0-9650d6f82a57" (UID: "abf4257b-2d15-421e-96c0-9650d6f82a57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:03.369021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.369000    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf4257b-2d15-421e-96c0-9650d6f82a57-kube-api-access-nt6t7" pod "abf4257b-2d15-421e-96c0-9650d6f82a57" (UID: "abf4257b-2d15-421e-96c0-9650d6f82a57"). InnerVolumeSpecName "kube-api-access-nt6t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:03.468190 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.468133    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt6t7\" (UniqueName: \"kubernetes.io/projected/abf4257b-2d15-421e-96c0-9650d6f82a57-kube-api-access-nt6t7\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.468190 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.468165    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf4257b-2d15-421e-96c0-9650d6f82a57-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.468190 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.468187    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf4257b-2d15-421e-96c0-9650d6f82a57-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.640260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.622889    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:03.770042 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.769997    4531 generic.go:356] "Generic (PLEG): container finished" podID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerID="7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691" exitCode=0
Nov 03 20:01:03.770333 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.770053    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"
Nov 03 20:01:03.770333 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.770081    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" event={"ID":"abf4257b-2d15-421e-96c0-9650d6f82a57","Type":"ContainerDied","Data":"7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691"}
Nov 03 20:01:03.770333 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.770142    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" event={"ID":"abf4257b-2d15-421e-96c0-9650d6f82a57","Type":"ContainerDied","Data":"7bfce0782243b943174e2c31dbff6b99cad262deaa2a8edef7eb13f97f315055"}
Nov 03 20:01:03.770333 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.770155    4531 scope.go:122] "RemoveContainer" containerID="7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691"
Nov 03 20:01:03.783710 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.783688    4531 scope.go:122] "RemoveContainer" containerID="7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691"
Nov 03 20:01:03.784004 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:01:03.783980    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691\": container with ID starting with 7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691 not found: ID does not exist" containerID="7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691"
Nov 03 20:01:03.784077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.784011    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691"} err="failed to get container status \"7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691\": rpc error: code = NotFound desc = could not find container \"7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691\": container with ID starting with 7020dbaf8bb9460950f5aa782bce6a92bf2462ba953857925dcddc0f8e256691 not found: ID does not exist"
Nov 03 20:01:03.794895 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.794861    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"]
Nov 03 20:01:03.800605 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:03.800579    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng"]
Nov 03 20:01:04.266132 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:04.266054    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bdb694/test-model-bdb694-kserve-98d86d685-7m9ng" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:04.462918 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:04.457560    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:04.864725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:04.864690    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" path="/var/lib/kubelet/pods/abf4257b-2d15-421e-96c0-9650d6f82a57/volumes"
Nov 03 20:01:05.227698 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:05.227575    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:06.065196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:06.059798    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:06.798235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:06.798155    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:07.576248 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:07.562880    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:08.265193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:08.237640    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:09.347945 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:09.347895    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:09.380380 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:09.380340    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"]
Nov 03 20:01:09.392076 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:09.392037    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"]
Nov 03 20:01:09.508480 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:09.508442    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:09.634311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:09.634272    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:10.361190 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:10.361130    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:11.060734 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:11.060628    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:11.758231 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:11.754549    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:12.369988 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:12.369940    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:13.139115 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:13.139074    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:13.815203 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:13.808389    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:14.102907 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.102804    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" podUID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" containerName="main" containerID="cri-o://40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574" gracePeriod=45
Nov 03 20:01:14.381386 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.381347    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk" podUID="496775e8-474f-4196-a144-4a8526f49bbe" containerName="payload-processing" containerID="cri-o://7c12a04aa5a395aa23b299e1a116a064c48390e482442e6e2bf2923f8f675dae" gracePeriod=25
Nov 03 20:01:14.392981 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.392923    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds" podUID="0951ddac-5219-4bc1-951b-21aece6dc501" containerName="payload-pre-processing" containerID="cri-o://719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1" gracePeriod=25
Nov 03 20:01:14.430909 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.429894    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:14.505243 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.504834    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 20:01:14.635400 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.635293    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tls-certs\" (UniqueName: \"kubernetes.io/secret/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tls-certs\") pod \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") "
Nov 03 20:01:14.635400 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.635384    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-model-cache\") pod \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") "
Nov 03 20:01:14.635657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.635561    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-kube-api-access-zbp29\" (UniqueName: \"kubernetes.io/projected/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-kube-api-access-zbp29\") pod \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") "
Nov 03 20:01:14.635657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.635604    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-dshm\") pod \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") "
Nov 03 20:01:14.635766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.635648    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-model-cache" pod "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" (UID: "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:14.635766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.635694    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-home\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-home\") pod \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") "
Nov 03 20:01:14.635841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.635805    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tmp-dir\") pod \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\" (UID: \"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46\") "
Nov 03 20:01:14.635987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.635959    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-home" pod "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" (UID: "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:14.636115 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.636080    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tmp-dir" pod "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" (UID: "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:14.636879 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.636856    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:14.636956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.636885    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:14.636956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.636925    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:14.637861 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.637828    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tls-certs" pod "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" (UID: "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:14.637861 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.637841    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-dshm" pod "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" (UID: "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:14.637979 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.637890    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-kube-api-access-zbp29" pod "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" (UID: "33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46"). InnerVolumeSpecName "kube-api-access-zbp29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:14.732905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.732861    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk" podUID="496775e8-474f-4196-a144-4a8526f49bbe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:14.733401 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.733373    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds" podUID="0951ddac-5219-4bc1-951b-21aece6dc501" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:14.738829 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.738804    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:14.738930 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.738833    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbp29\" (UniqueName: \"kubernetes.io/projected/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-kube-api-access-zbp29\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:14.738930 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.738844    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:14.940836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.940738    4531 generic.go:356] "Generic (PLEG): container finished" podID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" containerID="40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574" exitCode=0
Nov 03 20:01:14.940836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.940786    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"
Nov 03 20:01:14.940836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.940824    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" event={"ID":"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46","Type":"ContainerDied","Data":"40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574"}
Nov 03 20:01:14.941112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.940870    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk" event={"ID":"33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46","Type":"ContainerDied","Data":"4643731c97aa1cd3a3c63c808f23a1205e346b7baed2e490280f4ccdae109848"}
Nov 03 20:01:14.941112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.940892    4531 scope.go:122] "RemoveContainer" containerID="40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574"
Nov 03 20:01:14.953612 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.953394    4531 scope.go:122] "RemoveContainer" containerID="40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574"
Nov 03 20:01:14.953816 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:01:14.953793    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574\": container with ID starting with 40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574 not found: ID does not exist" containerID="40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574"
Nov 03 20:01:14.953918 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.953824    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574"} err="failed to get container status \"40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574\": rpc error: code = NotFound desc = could not find container \"40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574\": container with ID starting with 40e4950220e8d4d8ab1a12566777cdc438a87606fe3eefdad6f9ed282ef09574 not found: ID does not exist"
Nov 03 20:01:14.958979 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.958958    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"]
Nov 03 20:01:14.964484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:14.964461    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-332694/test-model-332694-kserve-865b9db95c-cjwdk"]
Nov 03 20:01:15.138381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:15.137999    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:15.646031 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:15.645994    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:16.862636 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:16.862605    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" path="/var/lib/kubelet/pods/33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46/volumes"
Nov 03 20:01:17.137867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:17.137819    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:17.288210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:17.286869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:17.927147 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:17.927109    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:18.564030 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:18.563985    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:19.262571 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:19.262162    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:19.905196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:19.904853    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:20.597007 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:20.596926    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:20.672501 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:01:20.672462    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:01:21.235420 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:21.235335    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:21.937532 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:21.937480    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:22.584522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:22.584467    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:22.858048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:22.857944    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:22.999526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:22.999492    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:23.108457 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:23.108372    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"]
Nov 03 20:01:23.108941 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:23.108906    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerName="istio-proxy" containerID="cri-o://a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab" gracePeriod=30
Nov 03 20:01:23.189019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:23.188970    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:23.338192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:23.332926    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:24.408717 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:24.408682    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:24.733503 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:24.733396    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds" podUID="0951ddac-5219-4bc1-951b-21aece6dc501" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:24.733503 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:24.733401    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk" podUID="496775e8-474f-4196-a144-4a8526f49bbe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:25.280647 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:25.280606    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:25.461882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:25.461845    4531 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:01:25.462387 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:25.461918    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:01:25.952369 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:25.952329    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:26.549865 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:26.549831    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:27.253806 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:27.253728    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:27.882195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:27.868680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:28.436873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.436844    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 20:01:28.473560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.473526    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"]
Nov 03 20:01:28.474757 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474730    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerName="cleanup"
Nov 03 20:01:28.474757 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474754    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474766    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474771    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474781    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474787    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474793    4531 cpu_manager.go:414] "containerMap: removing container" podUID="acc5d5e9-df4c-4644-b1a3-1577b98c9a80" containerName="revoke-keys"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474798    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc5d5e9-df4c-4644-b1a3-1577b98c9a80" containerName="revoke-keys"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474811    4531 cpu_manager.go:414] "containerMap: removing container" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerName="main"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474816    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerName="main"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474826    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474830    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474842    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474846    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474858    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474862    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474884    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474889    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc39eb6-3905-4203-b242-a28f6b12477a" containerName="cleanup"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474903    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerName="istio-proxy"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474908    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerName="istio-proxy"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474931    4531 cpu_manager.go:414] "containerMap: removing container" podUID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" containerName="main"
Nov 03 20:01:28.474987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.474937    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" containerName="main"
Nov 03 20:01:28.476048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.475085    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerName="istio-proxy"
Nov 03 20:01:28.476048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.475095    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerName="cleanup"
Nov 03 20:01:28.476048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.475104    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerName="cleanup"
Nov 03 20:01:28.476048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.475110    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerName="cleanup"
Nov 03 20:01:28.476048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.475118    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c4e3a6-a7ba-4fa3-a14e-e82edb1e4c46" containerName="main"
Nov 03 20:01:28.476048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.475126    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerName="cleanup"
Nov 03 20:01:28.476048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.475131    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerName="cleanup"
Nov 03 20:01:28.476048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.475136    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerName="cleanup"
Nov 03 20:01:28.476048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.475144    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc5d5e9-df4c-4644-b1a3-1577b98c9a80" containerName="revoke-keys"
Nov 03 20:01:28.476048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.475151    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf4257b-2d15-421e-96c0-9650d6f82a57" containerName="main"
Nov 03 20:01:28.476048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.475158    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerName="cleanup"
Nov 03 20:01:28.488447 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.488410    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"]
Nov 03 20:01:28.488643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.488534    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.492015 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.491986    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7a62d4-openshift-default-dockercfg-vb879\""
Nov 03 20:01:28.511042 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511002    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-data\") pod \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") "
Nov 03 20:01:28.511281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511263    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-socket\") pod \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") "
Nov 03 20:01:28.511379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511309    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-data" pod "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" (UID: "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:28.511458 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511382    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-envoy\") pod \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") "
Nov 03 20:01:28.511521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511494    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-certs\") pod \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") "
Nov 03 20:01:28.511578 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511552    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-socket" pod "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" (UID: "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:28.511639 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511613    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-kube-api-access-77fwr\" (UniqueName: \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-kube-api-access-77fwr\") pod \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") "
Nov 03 20:01:28.511703 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511681    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-podinfo\") pod \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") "
Nov 03 20:01:28.511819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511741    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-credential-socket\") pod \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") "
Nov 03 20:01:28.511819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511762    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-certs" pod "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" (UID: "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:28.511819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511813    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istiod-ca-cert\") pod \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") "
Nov 03 20:01:28.511992 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.511874    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-token\" (UniqueName: \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-token\") pod \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\" (UID: \"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8\") "
Nov 03 20:01:28.512059 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.512015    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-credential-socket" pod "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" (UID: "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:28.512302 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.512276    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istiod-ca-cert" pod "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" (UID: "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:28.512967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.512939    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:28.513063 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.512974    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:28.513063 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.512989    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:28.513063 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.513002    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:28.513063 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.513017    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:28.514986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.514224    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-kube-api-access-77fwr" pod "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" (UID: "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8"). InnerVolumeSpecName "kube-api-access-77fwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:28.514986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.514247    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-envoy" pod "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" (UID: "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:28.514986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.514462    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-token" pod "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" (UID: "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:28.515407 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.515387    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-podinfo" pod "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" (UID: "9a53a089-f3db-4d21-ab7b-f09d17aa4fc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:01:28.575336 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.559107    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.614731    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlhns\" (UniqueName: \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-kube-api-access-nlhns\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.614953    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-envoy\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.615010    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-credential-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.615037    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-podinfo\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.615074    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-data\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.615152    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-certs\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.638054    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73124dfd-65ad-4cc7-b44d-8520be810d29-istiod-ca-cert\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.638465    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-token\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.638527    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.638750    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.638770    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.638788    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:28.667533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.638805    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77fwr\" (UniqueName: \"kubernetes.io/projected/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8-kube-api-access-77fwr\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:28.741030 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.740978    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-envoy\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741048    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-credential-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741073    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-podinfo\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741107    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-data\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741195    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-certs\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741240    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73124dfd-65ad-4cc7-b44d-8520be810d29-istiod-ca-cert\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741540 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741363    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-token\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741540 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741405    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741540 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741510    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-credential-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741581    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-data\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741616    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-certs\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741636    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhns\" (UniqueName: \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-kube-api-access-nlhns\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.741862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.741810    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.742240 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.742218    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73124dfd-65ad-4cc7-b44d-8520be810d29-istiod-ca-cert\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.743584 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.743562    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-envoy\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.743816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.743797    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-podinfo\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.747915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.747893    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-token\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.748047 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.748030    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhns\" (UniqueName: \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-kube-api-access-nlhns\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:28.808420 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:28.808367    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:29.141101 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:01:29.141061    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73124dfd_65ad_4cc7_b44d_8520be810d29.slice/crio-bcb5350b2a252c3a33d5eb54ec1e61fc6f13ff959835a0910681f0658f3aa36e WatchSource:0}: Error finding container bcb5350b2a252c3a33d5eb54ec1e61fc6f13ff959835a0910681f0658f3aa36e: Status 404 returned error can't find the container with id bcb5350b2a252c3a33d5eb54ec1e61fc6f13ff959835a0910681f0658f3aa36e
Nov 03 20:01:29.142324 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.141830    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"]
Nov 03 20:01:29.145266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.145229    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:29.145364 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.145305    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:29.145430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.145361    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:29.166939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.166906    4531 generic.go:356] "Generic (PLEG): container finished" podID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" containerID="a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab" exitCode=0
Nov 03 20:01:29.167112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.166991    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"
Nov 03 20:01:29.167112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.166991    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" event={"ID":"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8","Type":"ContainerDied","Data":"a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab"}
Nov 03 20:01:29.167244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.167139    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj" event={"ID":"9a53a089-f3db-4d21-ab7b-f09d17aa4fc8","Type":"ContainerDied","Data":"603efe6f9a62fbc33fc6bac814a5dccc7c86f3eeb877968055726c7b98b364c1"}
Nov 03 20:01:29.167244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.167166    4531 scope.go:122] "RemoveContainer" containerID="a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab"
Nov 03 20:01:29.168704 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.168682    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" event={"ID":"73124dfd-65ad-4cc7-b44d-8520be810d29","Type":"ContainerStarted","Data":"bcb5350b2a252c3a33d5eb54ec1e61fc6f13ff959835a0910681f0658f3aa36e"}
Nov 03 20:01:29.197292 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.197268    4531 scope.go:122] "RemoveContainer" containerID="a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab"
Nov 03 20:01:29.197648 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:01:29.197623    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab\": container with ID starting with a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab not found: ID does not exist" containerID="a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab"
Nov 03 20:01:29.197696 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.197659    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab"} err="failed to get container status \"a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab\": rpc error: code = NotFound desc = could not find container \"a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab\": container with ID starting with a5855c44cc4f387087e748b404d2f9c26f3746db6ccd57f140b3895b56a856ab not found: ID does not exist"
Nov 03 20:01:29.225928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.224817    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"]
Nov 03 20:01:29.232081 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.232055    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-mkqrj"]
Nov 03 20:01:29.349830 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.347409    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:29.516364 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.516245    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:29.655733 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:29.655704    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:30.015274 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:30.014471    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:30.181664 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:30.181621    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" event={"ID":"73124dfd-65ad-4cc7-b44d-8520be810d29","Type":"ContainerStarted","Data":"73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083"}
Nov 03 20:01:30.203824 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:30.203760    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" podStartSLOduration=143.203745166 podStartE2EDuration="2m23.203745166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:30.198627868 +0000 UTC m=+1803.468751163" watchObservedRunningTime="2026-11-03 20:01:30.203745166 +0000 UTC m=+1803.473868460"
Nov 03 20:01:30.753201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:30.744114    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:30.814210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:30.813801    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:30.826197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:30.819213    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 20:01:30.826197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:30.819275    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" podUID="73124dfd-65ad-4cc7-b44d-8520be810d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 20:01:30.889805 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:30.889749    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a53a089-f3db-4d21-ab7b-f09d17aa4fc8" path="/var/lib/kubelet/pods/9a53a089-f3db-4d21-ab7b-f09d17aa4fc8/volumes"
Nov 03 20:01:31.411706 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:31.411644    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:31.809406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:31.809309    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 20:01:31.809406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:31.809373    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" podUID="73124dfd-65ad-4cc7-b44d-8520be810d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 20:01:32.196465 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:32.196424    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:32.468256 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:32.466059    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:01:32.494800 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:32.494775    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:01:32.808972 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:32.808883    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 20:01:32.808972 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:32.808945    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" podUID="73124dfd-65ad-4cc7-b44d-8520be810d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 20:01:32.849200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:32.845448    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:33.061992 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:33.061846    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:33.203042 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:33.203007    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:33.615466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:33.615432    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:33.811766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:33.811734    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 20:01:33.812027 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:33.812001    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" podUID="73124dfd-65ad-4cc7-b44d-8520be810d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 20:01:33.968017 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:33.967980    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"]
Nov 03 20:01:33.974870 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:33.974834    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"]
Nov 03 20:01:34.081240 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:34.081191    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:34.188431 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:34.188394    4531 scope.go:122] "RemoveContainer" containerID="526e66124d30ea0f79ba8d74fbd744387b6c6b4830c441659290fb876e458fa4"
Nov 03 20:01:34.205754 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:34.205726    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:34.436107 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:34.436071    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:34.746270 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:01:34.745241    4531 logging.go:55] [core] [Channel #6659 SubChannel #6660]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.172:50052", ServerName: "10.133.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.172:50052: connect: connection refused"
Nov 03 20:01:34.756194 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:01:34.750575    4531 logging.go:55] [core] [Channel #6661 SubChannel #6662]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.171:50052", ServerName: "10.133.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.171:50052: connect: connection refused"
Nov 03 20:01:34.812200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:34.811648    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 20:01:34.812200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:34.811715    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" podUID="73124dfd-65ad-4cc7-b44d-8520be810d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 20:01:35.210220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.205647    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 20:01:35.279943 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.279852    4531 generic.go:356] "Generic (PLEG): container finished" podID="496775e8-474f-4196-a144-4a8526f49bbe" containerID="7c12a04aa5a395aa23b299e1a116a064c48390e482442e6e2bf2923f8f675dae" exitCode=0
Nov 03 20:01:35.280275 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.280249    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk" event={"ID":"496775e8-474f-4196-a144-4a8526f49bbe","Type":"ContainerDied","Data":"7c12a04aa5a395aa23b299e1a116a064c48390e482442e6e2bf2923f8f675dae"}
Nov 03 20:01:35.282307 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.282276    4531 generic.go:356] "Generic (PLEG): container finished" podID="0951ddac-5219-4bc1-951b-21aece6dc501" containerID="719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1" exitCode=0
Nov 03 20:01:35.282445 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.282395    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds" event={"ID":"0951ddac-5219-4bc1-951b-21aece6dc501","Type":"ContainerDied","Data":"719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1"}
Nov 03 20:01:35.282564 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.282448    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds" event={"ID":"0951ddac-5219-4bc1-951b-21aece6dc501","Type":"ContainerDied","Data":"246b7d9da416abc6be6d9ada3ed69561aad96311230e705d8ff99e4536ce3b46"}
Nov 03 20:01:35.282564 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.282482    4531 scope.go:122] "RemoveContainer" containerID="719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1"
Nov 03 20:01:35.282740 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.282652    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"
Nov 03 20:01:35.310737 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.309114    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0951ddac-5219-4bc1-951b-21aece6dc501-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0951ddac-5219-4bc1-951b-21aece6dc501-plugins-config-volume\") pod \"0951ddac-5219-4bc1-951b-21aece6dc501\" (UID: \"0951ddac-5219-4bc1-951b-21aece6dc501\") "
Nov 03 20:01:35.310737 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.309384    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0951ddac-5219-4bc1-951b-21aece6dc501-kube-api-access-hg7hn\" (UniqueName: \"kubernetes.io/projected/0951ddac-5219-4bc1-951b-21aece6dc501-kube-api-access-hg7hn\") pod \"0951ddac-5219-4bc1-951b-21aece6dc501\" (UID: \"0951ddac-5219-4bc1-951b-21aece6dc501\") "
Nov 03 20:01:35.311606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.311581    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0951ddac-5219-4bc1-951b-21aece6dc501-plugins-config-volume" pod "0951ddac-5219-4bc1-951b-21aece6dc501" (UID: "0951ddac-5219-4bc1-951b-21aece6dc501"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:35.312654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.312618    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0951ddac-5219-4bc1-951b-21aece6dc501-kube-api-access-hg7hn" pod "0951ddac-5219-4bc1-951b-21aece6dc501" (UID: "0951ddac-5219-4bc1-951b-21aece6dc501"). InnerVolumeSpecName "kube-api-access-hg7hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:35.344242 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.343676    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:35.406326 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.391746    4531 scope.go:122] "RemoveContainer" containerID="719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1"
Nov 03 20:01:35.406326 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:01:35.396400    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1\": container with ID starting with 719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1 not found: ID does not exist" containerID="719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1"
Nov 03 20:01:35.406326 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.396468    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1"} err="failed to get container status \"719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1\": rpc error: code = NotFound desc = could not find container \"719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1\": container with ID starting with 719f7cdb7aeb7865c3b39aa07897b80ad74dd4aef29489c4ee6b2afca671d8d1 not found: ID does not exist"
Nov 03 20:01:35.476265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.476231    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg7hn\" (UniqueName: \"kubernetes.io/projected/0951ddac-5219-4bc1-951b-21aece6dc501-kube-api-access-hg7hn\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:35.476549 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.476526    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0951ddac-5219-4bc1-951b-21aece6dc501-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:35.480200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.479759    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 20:01:35.510224 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.510184    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"]
Nov 03 20:01:35.511310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511283    4531 cpu_manager.go:414] "containerMap: removing container" podUID="496775e8-474f-4196-a144-4a8526f49bbe" containerName="payload-processing"
Nov 03 20:01:35.511310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511309    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496775e8-474f-4196-a144-4a8526f49bbe" containerName="payload-processing"
Nov 03 20:01:35.511523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511346    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerName="cleanup"
Nov 03 20:01:35.511523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511354    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3159f7-387e-4e6f-ae38-d0bdabb6fcb7" containerName="cleanup"
Nov 03 20:01:35.511523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511377    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerName="cleanup"
Nov 03 20:01:35.511523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511384    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerName="cleanup"
Nov 03 20:01:35.511523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511412    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerName="cleanup"
Nov 03 20:01:35.511523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511419    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerName="cleanup"
Nov 03 20:01:35.511523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511454    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0951ddac-5219-4bc1-951b-21aece6dc501" containerName="payload-pre-processing"
Nov 03 20:01:35.511523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511461    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0951ddac-5219-4bc1-951b-21aece6dc501" containerName="payload-pre-processing"
Nov 03 20:01:35.511938 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511717    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7082dcc4-b73c-43f9-8f11-ff7c18ecb58c" containerName="cleanup"
Nov 03 20:01:35.511938 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511736    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6823572a-3a8a-410b-9821-3a2d3a06d2fb" containerName="cleanup"
Nov 03 20:01:35.511938 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511753    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="496775e8-474f-4196-a144-4a8526f49bbe" containerName="payload-processing"
Nov 03 20:01:35.511938 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.511776    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0951ddac-5219-4bc1-951b-21aece6dc501" containerName="payload-pre-processing"
Nov 03 20:01:35.517353 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.517326    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.526150 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.526113    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"]
Nov 03 20:01:35.577545 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.577511    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496775e8-474f-4196-a144-4a8526f49bbe-kube-api-access-72xp6\" (UniqueName: \"kubernetes.io/projected/496775e8-474f-4196-a144-4a8526f49bbe-kube-api-access-72xp6\") pod \"496775e8-474f-4196-a144-4a8526f49bbe\" (UID: \"496775e8-474f-4196-a144-4a8526f49bbe\") "
Nov 03 20:01:35.577772 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.577630    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496775e8-474f-4196-a144-4a8526f49bbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496775e8-474f-4196-a144-4a8526f49bbe-plugins-config-volume\") pod \"496775e8-474f-4196-a144-4a8526f49bbe\" (UID: \"496775e8-474f-4196-a144-4a8526f49bbe\") "
Nov 03 20:01:35.578123 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.578095    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66385a3-91b7-4e59-afbd-d6a57110a267-istiod-ca-cert\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.578294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.578154    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-token\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.578294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.578225    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp54h\" (UniqueName: \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-kube-api-access-jp54h\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.578294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.578253    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-podinfo\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.578494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.578430    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-credential-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.578494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.578453    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-certs\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.578614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.578548    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.578614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.578564    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-envoy\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.578614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.578599    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-data\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.579406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.579377    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496775e8-474f-4196-a144-4a8526f49bbe-plugins-config-volume" pod "496775e8-474f-4196-a144-4a8526f49bbe" (UID: "496775e8-474f-4196-a144-4a8526f49bbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:35.582512 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.582484    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496775e8-474f-4196-a144-4a8526f49bbe-kube-api-access-72xp6" pod "496775e8-474f-4196-a144-4a8526f49bbe" (UID: "496775e8-474f-4196-a144-4a8526f49bbe"). InnerVolumeSpecName "kube-api-access-72xp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:35.626866 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.626832    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"]
Nov 03 20:01:35.632186 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.632136    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds"]
Nov 03 20:01:35.680079 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680044    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-certs\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.680314 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680187    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.680314 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680217    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-envoy\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.680314 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680269    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-data\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.680464 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680390    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66385a3-91b7-4e59-afbd-d6a57110a267-istiod-ca-cert\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.680464 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680452    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-token\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.680546 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680518    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp54h\" (UniqueName: \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-kube-api-access-jp54h\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.680589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680540    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-certs\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.680589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680556    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-podinfo\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.680768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680742    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.680873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680852    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-credential-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.680989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680971    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72xp6\" (UniqueName: \"kubernetes.io/projected/496775e8-474f-4196-a144-4a8526f49bbe-kube-api-access-72xp6\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:35.681071 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.680991    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/496775e8-474f-4196-a144-4a8526f49bbe-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:35.681159 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.681138    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66385a3-91b7-4e59-afbd-d6a57110a267-istiod-ca-cert\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.681266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.681144    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-data\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.681266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.681246    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-credential-socket\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.682709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.682691    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-envoy\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.683098 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.683080    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-podinfo\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.688544 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.688522    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-token\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.688686 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.688669    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp54h\" (UniqueName: \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-kube-api-access-jp54h\") pod \"e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") " pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:35.761518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.761467    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk" podUID="496775e8-474f-4196-a144-4a8526f49bbe" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.172:50052\" within 1s: context deadline exceeded"
Nov 03 20:01:35.762021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.761576    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 20:01:35.762021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.761467    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bdb694-6c4ff8cb5f-qs6ds" podUID="0951ddac-5219-4bc1-951b-21aece6dc501" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.171:50052\" within 1s: context deadline exceeded"
Nov 03 20:01:35.762021 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:01:35.761639    4531 logging.go:55] [core] [Channel #6659 SubChannel #6660]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.172:50052", ServerName: "10.133.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.172:50052: operation was canceled"
Nov 03 20:01:35.762021 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:01:35.761639    4531 logging.go:55] [core] [Channel #6661 SubChannel #6662]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.171:50052", ServerName: "10.133.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.171:50052: operation was canceled"
Nov 03 20:01:35.809241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.809108    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 20:01:35.809241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.809167    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" podUID="73124dfd-65ad-4cc7-b44d-8520be810d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 20:01:35.858198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:35.857557    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:36.161205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.151771    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:36.367274 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.362496    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"]
Nov 03 20:01:36.376352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.376308    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk" event={"ID":"496775e8-474f-4196-a144-4a8526f49bbe","Type":"ContainerDied","Data":"106395ad7c7df3e35e90c934fc43506710ef169e271fa69204ec4e3b4863d4e9"}
Nov 03 20:01:36.376483 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.376390    4531 scope.go:122] "RemoveContainer" containerID="7c12a04aa5a395aa23b299e1a116a064c48390e482442e6e2bf2923f8f675dae"
Nov 03 20:01:36.377263 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.377237    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"
Nov 03 20:01:36.386748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.386697    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:36.386867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.386807    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:36.386867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.386855    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:36.417724 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.417696    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"]
Nov 03 20:01:36.423223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.423198    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bdb694-7f64dd85b7-4d7fk"]
Nov 03 20:01:36.869942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.869839    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0951ddac-5219-4bc1-951b-21aece6dc501" path="/var/lib/kubelet/pods/0951ddac-5219-4bc1-951b-21aece6dc501/volumes"
Nov 03 20:01:36.870758 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.870733    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496775e8-474f-4196-a144-4a8526f49bbe" path="/var/lib/kubelet/pods/496775e8-474f-4196-a144-4a8526f49bbe/volumes"
Nov 03 20:01:36.910899 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:36.910863    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:37.117995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:37.117959    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:37.118828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:37.118799    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:37.121483 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:37.121408    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:01:37.386834 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:37.386793    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" event={"ID":"b66385a3-91b7-4e59-afbd-d6a57110a267","Type":"ContainerStarted","Data":"b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7"}
Nov 03 20:01:37.387013 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:37.386866    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" event={"ID":"b66385a3-91b7-4e59-afbd-d6a57110a267","Type":"ContainerStarted","Data":"434ec3bdb17560c286e376a3c53c925658a17d741b57c116c2d4f0f159e07162"}
Nov 03 20:01:37.409633 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:37.409579    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" podStartSLOduration=217.409565218 podStartE2EDuration="3m37.409565218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:37.404519775 +0000 UTC m=+1810.674643070" watchObservedRunningTime="2026-11-03 20:01:37.409565218 +0000 UTC m=+1810.679688513"
Nov 03 20:01:37.568205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:37.558893    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:37.861565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:37.861433    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:37.863081 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:37.863043    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:01:37.863283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:37.863104    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 20:01:38.341104 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:38.325006    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:38.806994 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:38.806948    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"]
Nov 03 20:01:38.826723 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:38.826688    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:38.999386 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:38.999339    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd" podUID="6551edf5-a4a6-4b4d-bb17-1b32b335c58e" containerName="payload-pre-processing" containerID="cri-o://003266267a5342a6d8a2c2ff281030e9b303fb99e2bbe031ce333a9f90e313dd" gracePeriod=25
Nov 03 20:01:39.004114 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.003577    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:01:39.004114 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.003650    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 20:01:39.032481 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.032014    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr" podUID="db439ef2-1c8e-4969-8c07-78c7ec87c3d9" containerName="payload-processing" containerID="cri-o://11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6" gracePeriod=25
Nov 03 20:01:39.190353 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.190294    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:39.473154 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.473103    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"]
Nov 03 20:01:39.492941 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.488425    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"]
Nov 03 20:01:39.492941 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.488573    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:39.492941 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.492638    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:01:39.493292 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.493080    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:01:39.593550 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.593505    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5nn\" (UniqueName: \"kubernetes.io/projected/baa40e8f-14ac-4070-8d1d-1276da8d481a-kube-api-access-zw5nn\") pod \"maas-api-revoke-sub-07655733-vdx7s\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:39.593965 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.593940    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07655733-vdx7s\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:39.594215 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.594155    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07655733-vdx7s\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:39.699226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.698248    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5nn\" (UniqueName: \"kubernetes.io/projected/baa40e8f-14ac-4070-8d1d-1276da8d481a-kube-api-access-zw5nn\") pod \"maas-api-revoke-sub-07655733-vdx7s\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:39.699226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.698464    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07655733-vdx7s\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:39.699226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.698534    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07655733-vdx7s\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:39.701976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.701940    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07655733-vdx7s\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:39.709678 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.709643    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07655733-vdx7s\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:39.711374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.711347    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5nn\" (UniqueName: \"kubernetes.io/projected/baa40e8f-14ac-4070-8d1d-1276da8d481a-kube-api-access-zw5nn\") pod \"maas-api-revoke-sub-07655733-vdx7s\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:39.819020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.818919    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:39.859928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.859882    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:01:39.860121 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.859951    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 20:01:39.963215 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:39.957100    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:40.374768 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:01:40.374733    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa40e8f_14ac_4070_8d1d_1276da8d481a.slice/crio-cd0e08b7ebcbd26ad018898190b78b22aaced576265b23436dcc306e7e93d5e5 WatchSource:0}: Error finding container cd0e08b7ebcbd26ad018898190b78b22aaced576265b23436dcc306e7e93d5e5: Status 404 returned error can't find the container with id cd0e08b7ebcbd26ad018898190b78b22aaced576265b23436dcc306e7e93d5e5
Nov 03 20:01:40.377917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:40.377886    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"]
Nov 03 20:01:40.564234 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:40.564164    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s" event={"ID":"baa40e8f-14ac-4070-8d1d-1276da8d481a","Type":"ContainerStarted","Data":"cd0e08b7ebcbd26ad018898190b78b22aaced576265b23436dcc306e7e93d5e5"}
Nov 03 20:01:40.754196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:40.748787    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:40.879371 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:40.879328    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:01:40.879596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:40.879385    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 20:01:41.440200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:41.429398    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd" podUID="6551edf5-a4a6-4b4d-bb17-1b32b335c58e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:41.452195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:41.449624    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr" podUID="db439ef2-1c8e-4969-8c07-78c7ec87c3d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:41.501198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:41.493860    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:41.698244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:41.692256    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s" event={"ID":"baa40e8f-14ac-4070-8d1d-1276da8d481a","Type":"ContainerStarted","Data":"75d417d35ba14c58c40faf4172c7786ad40020f806213662932f3b92cb9244fa"}
Nov 03 20:01:41.780484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:41.772373    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s" podStartSLOduration=2.7723495700000003 podStartE2EDuration="2.77234957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:41.760972254 +0000 UTC m=+1815.031095550" watchObservedRunningTime="2026-11-03 20:01:41.77234957 +0000 UTC m=+1815.042472867"
Nov 03 20:01:41.844296 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:41.844128    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:41.859628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:41.859595    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:01:41.859934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:41.859905    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 20:01:42.096241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:42.087756    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:42.548199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:42.547103    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:42.719593 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:42.719559    4531 generic.go:356] "Generic (PLEG): container finished" podID="baa40e8f-14ac-4070-8d1d-1276da8d481a" containerID="75d417d35ba14c58c40faf4172c7786ad40020f806213662932f3b92cb9244fa" exitCode=0
Nov 03 20:01:42.720066 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:42.719797    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s" event={"ID":"baa40e8f-14ac-4070-8d1d-1276da8d481a","Type":"ContainerDied","Data":"75d417d35ba14c58c40faf4172c7786ad40020f806213662932f3b92cb9244fa"}
Nov 03 20:01:42.863254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:42.862656    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:01:42.863254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:42.862725    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 20:01:43.397743 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:43.397703    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:43.863293 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:43.859921    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:01:43.863293 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:43.859995    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Nov 03 20:01:44.189877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.189846    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:44.224486 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.224448    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa40e8f-14ac-4070-8d1d-1276da8d481a-kube-api-access-zw5nn\" (UniqueName: \"kubernetes.io/projected/baa40e8f-14ac-4070-8d1d-1276da8d481a-kube-api-access-zw5nn\") pod \"baa40e8f-14ac-4070-8d1d-1276da8d481a\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") "
Nov 03 20:01:44.224694 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.224556    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-openshift-ca\") pod \"baa40e8f-14ac-4070-8d1d-1276da8d481a\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") "
Nov 03 20:01:44.225350 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.225303    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-openshift-ca" pod "baa40e8f-14ac-4070-8d1d-1276da8d481a" (UID: "baa40e8f-14ac-4070-8d1d-1276da8d481a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:44.225510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.225379    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-xks-ca\") pod \"baa40e8f-14ac-4070-8d1d-1276da8d481a\" (UID: \"baa40e8f-14ac-4070-8d1d-1276da8d481a\") "
Nov 03 20:01:44.226765 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.226735    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:44.229185 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.228729    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa40e8f-14ac-4070-8d1d-1276da8d481a-kube-api-access-zw5nn" pod "baa40e8f-14ac-4070-8d1d-1276da8d481a" (UID: "baa40e8f-14ac-4070-8d1d-1276da8d481a"). InnerVolumeSpecName "kube-api-access-zw5nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:44.231256 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.231214    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-xks-ca" pod "baa40e8f-14ac-4070-8d1d-1276da8d481a" (UID: "baa40e8f-14ac-4070-8d1d-1276da8d481a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:44.329147 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.329086    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa40e8f-14ac-4070-8d1d-1276da8d481a-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:44.329147 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.329129    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw5nn\" (UniqueName: \"kubernetes.io/projected/baa40e8f-14ac-4070-8d1d-1276da8d481a-kube-api-access-zw5nn\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:44.419341 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.409222    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:44.923939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.923905    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"
Nov 03 20:01:44.923939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.923922    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s" event={"ID":"baa40e8f-14ac-4070-8d1d-1276da8d481a","Type":"ContainerDied","Data":"cd0e08b7ebcbd26ad018898190b78b22aaced576265b23436dcc306e7e93d5e5"}
Nov 03 20:01:44.924609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.923992    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd0e08b7ebcbd26ad018898190b78b22aaced576265b23436dcc306e7e93d5e5"
Nov 03 20:01:44.951504 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.951470    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:44.951999 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.951971    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:44.954269 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:44.954242    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:01:45.273929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:45.273831    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:46.042348 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:46.042312    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:46.862527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:46.848416    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:47.620159 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:47.617923    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:48.371223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:48.366847    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:49.127434 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:49.127391    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:49.963942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:49.963904    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:50.099339 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:50.099202    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"]
Nov 03 20:01:50.102029 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:50.099886    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy" containerID="cri-o://12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305" gracePeriod=30
Nov 03 20:01:50.308801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:50.308700    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"]
Nov 03 20:01:50.387589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:50.387548    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:50.757725 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:01:50.757680    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:01:51.001975 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:51.001534    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:51.409356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:51.409313    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd" podUID="6551edf5-a4a6-4b4d-bb17-1b32b335c58e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:51.409844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:51.409429    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr" podUID="db439ef2-1c8e-4969-8c07-78c7ec87c3d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:51.563860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:51.563823    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:51.715828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:51.715738    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:51.841683 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:51.841647    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:52.393999 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:52.393955    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:52.531937 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:52.531903    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:52.691329 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:52.691237    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:53.018417 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:53.018336    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:53.455652 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:53.455586    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:54.447199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.446285    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:54.458309 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.451590    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl" podUID="cd542f86-cf28-4141-b7f7-b0661226c455" containerName="main" containerID="cri-o://3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79" gracePeriod=45
Nov 03 20:01:54.565638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.565601    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl" podUID="cd542f86-cf28-4141-b7f7-b0661226c455" containerName="main" probeResult="failure" output="Get \"https://10.133.0.195:8000/ready\": dial tcp 10.133.0.195:8000: connect: connection refused"
Nov 03 20:01:54.810289 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.810260    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 20:01:54.937794 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.937756    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-home\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-home\") pod \"cd542f86-cf28-4141-b7f7-b0661226c455\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") "
Nov 03 20:01:54.938015 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.937819    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd542f86-cf28-4141-b7f7-b0661226c455-kube-api-access-vbhqs\" (UniqueName: \"kubernetes.io/projected/cd542f86-cf28-4141-b7f7-b0661226c455-kube-api-access-vbhqs\") pod \"cd542f86-cf28-4141-b7f7-b0661226c455\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") "
Nov 03 20:01:54.938015 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.937937    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd542f86-cf28-4141-b7f7-b0661226c455-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd542f86-cf28-4141-b7f7-b0661226c455-tls-certs\") pod \"cd542f86-cf28-4141-b7f7-b0661226c455\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") "
Nov 03 20:01:54.938161 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.938015    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-home" pod "cd542f86-cf28-4141-b7f7-b0661226c455" (UID: "cd542f86-cf28-4141-b7f7-b0661226c455"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:54.938161 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.938074    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-model-cache\") pod \"cd542f86-cf28-4141-b7f7-b0661226c455\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") "
Nov 03 20:01:54.938343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.938240    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-tmp-dir\") pod \"cd542f86-cf28-4141-b7f7-b0661226c455\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") "
Nov 03 20:01:54.938343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.938277    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-model-cache" pod "cd542f86-cf28-4141-b7f7-b0661226c455" (UID: "cd542f86-cf28-4141-b7f7-b0661226c455"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:54.938343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.938335    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-dshm\") pod \"cd542f86-cf28-4141-b7f7-b0661226c455\" (UID: \"cd542f86-cf28-4141-b7f7-b0661226c455\") "
Nov 03 20:01:54.938498 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.938476    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-tmp-dir" pod "cd542f86-cf28-4141-b7f7-b0661226c455" (UID: "cd542f86-cf28-4141-b7f7-b0661226c455"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:54.939551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.939520    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:54.939551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.939552    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:54.939748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.939569    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:54.940343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.940317    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd542f86-cf28-4141-b7f7-b0661226c455-kube-api-access-vbhqs" pod "cd542f86-cf28-4141-b7f7-b0661226c455" (UID: "cd542f86-cf28-4141-b7f7-b0661226c455"). InnerVolumeSpecName "kube-api-access-vbhqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:54.940429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.940403    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-dshm" pod "cd542f86-cf28-4141-b7f7-b0661226c455" (UID: "cd542f86-cf28-4141-b7f7-b0661226c455"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:54.940474 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:54.940452    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd542f86-cf28-4141-b7f7-b0661226c455-tls-certs" pod "cd542f86-cf28-4141-b7f7-b0661226c455" (UID: "cd542f86-cf28-4141-b7f7-b0661226c455"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:55.041597 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.041508    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd542f86-cf28-4141-b7f7-b0661226c455-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.041597 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.041542    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd542f86-cf28-4141-b7f7-b0661226c455-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.041597 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.041555    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbhqs\" (UniqueName: \"kubernetes.io/projected/cd542f86-cf28-4141-b7f7-b0661226c455-kube-api-access-vbhqs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.183615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.183582    4531 generic.go:356] "Generic (PLEG): container finished" podID="cd542f86-cf28-4141-b7f7-b0661226c455" containerID="3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79" exitCode=0
Nov 03 20:01:55.183815 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.183627    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"
Nov 03 20:01:55.183815 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.183641    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl" event={"ID":"cd542f86-cf28-4141-b7f7-b0661226c455","Type":"ContainerDied","Data":"3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79"}
Nov 03 20:01:55.183815 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.183705    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl" event={"ID":"cd542f86-cf28-4141-b7f7-b0661226c455","Type":"ContainerDied","Data":"8b93d99d0c15da2442a3e352652c70a483bb2eedb0dcb6aa268a865c81ca4d29"}
Nov 03 20:01:55.183815 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.183729    4531 scope.go:122] "RemoveContainer" containerID="3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79"
Nov 03 20:01:55.219292 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.219246    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"]
Nov 03 20:01:55.225257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.225224    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ffe132/test-model-ffe132-kserve-5cb4d484f7-sjgxl"]
Nov 03 20:01:55.257956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.257919    4531 scope.go:122] "RemoveContainer" containerID="3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79"
Nov 03 20:01:55.258455 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:01:55.258423    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79\": container with ID starting with 3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79 not found: ID does not exist" containerID="3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79"
Nov 03 20:01:55.258589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.258468    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79"} err="failed to get container status \"3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79\": rpc error: code = NotFound desc = could not find container \"3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79\": container with ID starting with 3652b4dfae0952ec8f12ef74185d9fe506b7ea8686d3f67a60fcca6a39394e79 not found: ID does not exist"
Nov 03 20:01:55.408802 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.408659    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:55.475193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.470690    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"]
Nov 03 20:01:55.552213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.543004    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerName="istio-proxy" containerID="cri-o://ac9467ce31127da3b59ab80b4abd00f54c3c38e37a395bc35d8f752eab378170" gracePeriod=30
Nov 03 20:01:55.635123 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.635094    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 20:01:55.653043 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.653009    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-envoy\") pod \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") "
Nov 03 20:01:55.653275 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.653146    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-kube-api-access-65jcw\" (UniqueName: \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-kube-api-access-65jcw\") pod \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") "
Nov 03 20:01:55.653367 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.653280    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-podinfo\") pod \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") "
Nov 03 20:01:55.653367 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.653306    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-socket\") pod \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") "
Nov 03 20:01:55.653367 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.653343    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istiod-ca-cert\") pod \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") "
Nov 03 20:01:55.653545 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.653444    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-credential-socket\") pod \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") "
Nov 03 20:01:55.653545 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.653481    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-certs\") pod \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") "
Nov 03 20:01:55.653658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.653557    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-data\") pod \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") "
Nov 03 20:01:55.653658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.653587    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-token\" (UniqueName: \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-token\") pod \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\" (UID: \"d5db9016-7dd6-4881-8e89-2bb9751a23c1\") "
Nov 03 20:01:55.654143 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.654097    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-socket" pod "d5db9016-7dd6-4881-8e89-2bb9751a23c1" (UID: "d5db9016-7dd6-4881-8e89-2bb9751a23c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:55.654340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.654144    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-credential-socket" pod "d5db9016-7dd6-4881-8e89-2bb9751a23c1" (UID: "d5db9016-7dd6-4881-8e89-2bb9751a23c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:55.654489 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.654467    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.654560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.654495    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istiod-ca-cert" pod "d5db9016-7dd6-4881-8e89-2bb9751a23c1" (UID: "d5db9016-7dd6-4881-8e89-2bb9751a23c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:55.654764 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.654740    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-certs" pod "d5db9016-7dd6-4881-8e89-2bb9751a23c1" (UID: "d5db9016-7dd6-4881-8e89-2bb9751a23c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:55.654891 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.654868    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-data" pod "d5db9016-7dd6-4881-8e89-2bb9751a23c1" (UID: "d5db9016-7dd6-4881-8e89-2bb9751a23c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:55.658930 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.658905    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-kube-api-access-65jcw" pod "d5db9016-7dd6-4881-8e89-2bb9751a23c1" (UID: "d5db9016-7dd6-4881-8e89-2bb9751a23c1"). InnerVolumeSpecName "kube-api-access-65jcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:55.659930 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.659608    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-envoy" pod "d5db9016-7dd6-4881-8e89-2bb9751a23c1" (UID: "d5db9016-7dd6-4881-8e89-2bb9751a23c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:55.659930 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.659716    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-podinfo" pod "d5db9016-7dd6-4881-8e89-2bb9751a23c1" (UID: "d5db9016-7dd6-4881-8e89-2bb9751a23c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:01:55.679890 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.679841    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-token" pod "d5db9016-7dd6-4881-8e89-2bb9751a23c1" (UID: "d5db9016-7dd6-4881-8e89-2bb9751a23c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:55.756695 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.756649    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.756695 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.756687    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.756695 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.756696    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.757036 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.756708    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65jcw\" (UniqueName: \"kubernetes.io/projected/d5db9016-7dd6-4881-8e89-2bb9751a23c1-kube-api-access-65jcw\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.757036 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.756719    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.757036 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.756727    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5db9016-7dd6-4881-8e89-2bb9751a23c1-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.757036 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.756735    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.757036 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:55.756743    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5db9016-7dd6-4881-8e89-2bb9751a23c1-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:56.197142 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.197103    4531 generic.go:356] "Generic (PLEG): container finished" podID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerID="12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305" exitCode=0
Nov 03 20:01:56.197398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.197213    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"
Nov 03 20:01:56.197398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.197225    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" event={"ID":"d5db9016-7dd6-4881-8e89-2bb9751a23c1","Type":"ContainerDied","Data":"12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305"}
Nov 03 20:01:56.197398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.197285    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb" event={"ID":"d5db9016-7dd6-4881-8e89-2bb9751a23c1","Type":"ContainerDied","Data":"0b3d1f5a51226cc8d2f40d362eb6730a90098482518e39d7f213a0bf4ca27703"}
Nov 03 20:01:56.197398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.197307    4531 scope.go:122] "RemoveContainer" containerID="12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305"
Nov 03 20:01:56.210521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.210266    4531 scope.go:122] "RemoveContainer" containerID="12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305"
Nov 03 20:01:56.210716 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:01:56.210683    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305\": container with ID starting with 12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305 not found: ID does not exist" containerID="12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305"
Nov 03 20:01:56.210835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.210729    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305"} err="failed to get container status \"12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305\": rpc error: code = NotFound desc = could not find container \"12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305\": container with ID starting with 12c16fdecd422cd2afa1affcf627e54e1a2d12765901a8882050ca3ead4a3305 not found: ID does not exist"
Nov 03 20:01:56.224122 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.224094    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"]
Nov 03 20:01:56.260201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.254224    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-z76xb"]
Nov 03 20:01:56.268193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.263768    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:56.863301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.863269    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd542f86-cf28-4141-b7f7-b0661226c455" path="/var/lib/kubelet/pods/cd542f86-cf28-4141-b7f7-b0661226c455/volumes"
Nov 03 20:01:56.863937 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:56.863920    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" path="/var/lib/kubelet/pods/d5db9016-7dd6-4881-8e89-2bb9751a23c1/volumes"
Nov 03 20:01:57.143430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:57.143393    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:58.085201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:58.077587    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:58.207806 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:58.207773    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:58.894794 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:58.894748    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:59.064113 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:59.064082    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:59.208198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:59.206848    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:59.569331 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:59.569226    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"]
Nov 03 20:01:59.584504 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:59.584460    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerName="istio-proxy" containerID="cri-o://73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434" gracePeriod=30
Nov 03 20:01:59.684353 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:59.651866    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:59.957920 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:59.957192    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"]
Nov 03 20:01:59.957920 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:59.957450    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerName="istio-proxy" containerID="cri-o://cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b" gracePeriod=30
Nov 03 20:01:59.983845 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:59.983809    4531 generic.go:356] "Generic (PLEG): container finished" podID="6551edf5-a4a6-4b4d-bb17-1b32b335c58e" containerID="003266267a5342a6d8a2c2ff281030e9b303fb99e2bbe031ce333a9f90e313dd" exitCode=0
Nov 03 20:01:59.984113 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:01:59.984082    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd" event={"ID":"6551edf5-a4a6-4b4d-bb17-1b32b335c58e","Type":"ContainerDied","Data":"003266267a5342a6d8a2c2ff281030e9b303fb99e2bbe031ce333a9f90e313dd"}
Nov 03 20:02:00.055058 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.055025    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 20:02:00.114992 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.114949    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:00.150138 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.150104    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-plugins-config-volume\") pod \"6551edf5-a4a6-4b4d-bb17-1b32b335c58e\" (UID: \"6551edf5-a4a6-4b4d-bb17-1b32b335c58e\") "
Nov 03 20:02:00.150411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.150392    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-kube-api-access-2zmt5\" (UniqueName: \"kubernetes.io/projected/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-kube-api-access-2zmt5\") pod \"6551edf5-a4a6-4b4d-bb17-1b32b335c58e\" (UID: \"6551edf5-a4a6-4b4d-bb17-1b32b335c58e\") "
Nov 03 20:02:00.150607 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.150573    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-plugins-config-volume" pod "6551edf5-a4a6-4b4d-bb17-1b32b335c58e" (UID: "6551edf5-a4a6-4b4d-bb17-1b32b335c58e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:00.151615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.151587    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:00.153220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.153190    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-kube-api-access-2zmt5" pod "6551edf5-a4a6-4b4d-bb17-1b32b335c58e" (UID: "6551edf5-a4a6-4b4d-bb17-1b32b335c58e"). InnerVolumeSpecName "kube-api-access-2zmt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:00.198999 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.198931    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 20:02:00.242665 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.242624    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:00.252975 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.252943    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-kube-api-access-b8sjz\" (UniqueName: \"kubernetes.io/projected/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-kube-api-access-b8sjz\") pod \"db439ef2-1c8e-4969-8c07-78c7ec87c3d9\" (UID: \"db439ef2-1c8e-4969-8c07-78c7ec87c3d9\") "
Nov 03 20:02:00.253146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.253045    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-plugins-config-volume\") pod \"db439ef2-1c8e-4969-8c07-78c7ec87c3d9\" (UID: \"db439ef2-1c8e-4969-8c07-78c7ec87c3d9\") "
Nov 03 20:02:00.253478 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.253454    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-plugins-config-volume" pod "db439ef2-1c8e-4969-8c07-78c7ec87c3d9" (UID: "db439ef2-1c8e-4969-8c07-78c7ec87c3d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:00.254015 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.253992    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zmt5\" (UniqueName: \"kubernetes.io/projected/6551edf5-a4a6-4b4d-bb17-1b32b335c58e-kube-api-access-2zmt5\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:00.254213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.254020    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:00.255771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.255743    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-kube-api-access-b8sjz" pod "db439ef2-1c8e-4969-8c07-78c7ec87c3d9" (UID: "db439ef2-1c8e-4969-8c07-78c7ec87c3d9"). InnerVolumeSpecName "kube-api-access-b8sjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:00.356434 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.356390    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8sjz\" (UniqueName: \"kubernetes.io/projected/db439ef2-1c8e-4969-8c07-78c7ec87c3d9-kube-api-access-b8sjz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:00.459284 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.447082    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:00.678081 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:00.678044    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:01.091255 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.071730    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:01.274830 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.274145    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"]
Nov 03 20:02:01.286251 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.282384    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"]
Nov 03 20:02:01.288907 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.288880    4531 generic.go:356] "Generic (PLEG): container finished" podID="db439ef2-1c8e-4969-8c07-78c7ec87c3d9" containerID="11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6" exitCode=0
Nov 03 20:02:01.289126 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.289106    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"
Nov 03 20:02:01.291395 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.289105    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr" event={"ID":"db439ef2-1c8e-4969-8c07-78c7ec87c3d9","Type":"ContainerDied","Data":"11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6"}
Nov 03 20:02:01.291395 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.290366    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr" event={"ID":"db439ef2-1c8e-4969-8c07-78c7ec87c3d9","Type":"ContainerDied","Data":"94e5f64318d4521826373717a2a7c964d0b95e805898b92f6c8f90826e14fb19"}
Nov 03 20:02:01.291395 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.290403    4531 scope.go:122] "RemoveContainer" containerID="11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6"
Nov 03 20:02:01.296564 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.296536    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd" event={"ID":"6551edf5-a4a6-4b4d-bb17-1b32b335c58e","Type":"ContainerDied","Data":"6b69122991b5ff18d9f8d4f8c1f86ed4c8a3df5f5c8e672f573f6b5e58998be2"}
Nov 03 20:02:01.296662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.296644    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"
Nov 03 20:02:01.312271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.312240    4531 generic.go:356] "Generic (PLEG): container finished" podID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerID="ac9467ce31127da3b59ab80b4abd00f54c3c38e37a395bc35d8f752eab378170" exitCode=0
Nov 03 20:02:01.312515 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.312494    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" event={"ID":"6e3fc4ac-760e-444b-a9eb-967f4d443b8e","Type":"ContainerDied","Data":"ac9467ce31127da3b59ab80b4abd00f54c3c38e37a395bc35d8f752eab378170"}
Nov 03 20:02:01.321041 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.321015    4531 scope.go:122] "RemoveContainer" containerID="11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6"
Nov 03 20:02:01.321370 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:01.321347    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6\": container with ID starting with 11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6 not found: ID does not exist" containerID="11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6"
Nov 03 20:02:01.321444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.321378    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6"} err="failed to get container status \"11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6\": rpc error: code = NotFound desc = could not find container \"11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6\": container with ID starting with 11f606b60203df5b8608f48b9c10fc7bad00f4d27856bf974f60402032c1f0c6 not found: ID does not exist"
Nov 03 20:02:01.321444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.321397    4531 scope.go:122] "RemoveContainer" containerID="003266267a5342a6d8a2c2ff281030e9b303fb99e2bbe031ce333a9f90e313dd"
Nov 03 20:02:01.344445 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.344310    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:01.349016 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.348867    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"]
Nov 03 20:02:01.355573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.355543    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-332694-56f87fc569-q5qgr"]
Nov 03 20:02:01.361551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.361521    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"]
Nov 03 20:02:01.368579 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.368388    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-332694-67bfcd58ff-nqdcd"]
Nov 03 20:02:01.374763 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.374741    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 20:02:01.445322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.445285    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-podinfo\") pod \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") "
Nov 03 20:02:01.445506 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.445344    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-credential-socket\") pod \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") "
Nov 03 20:02:01.445562 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.445530    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istiod-ca-cert\") pod \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") "
Nov 03 20:02:01.445614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.445566    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-data\") pod \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") "
Nov 03 20:02:01.445676 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.445647    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-envoy\") pod \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") "
Nov 03 20:02:01.445733 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.445713    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-socket\") pod \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") "
Nov 03 20:02:01.445796 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.445647    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-credential-socket" pod "6e3fc4ac-760e-444b-a9eb-967f4d443b8e" (UID: "6e3fc4ac-760e-444b-a9eb-967f4d443b8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:01.445951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.445923    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-data" pod "6e3fc4ac-760e-444b-a9eb-967f4d443b8e" (UID: "6e3fc4ac-760e-444b-a9eb-967f4d443b8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:01.446028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.445932    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istiod-ca-cert" pod "6e3fc4ac-760e-444b-a9eb-967f4d443b8e" (UID: "6e3fc4ac-760e-444b-a9eb-967f4d443b8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:01.446028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.445964    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-socket" pod "6e3fc4ac-760e-444b-a9eb-967f4d443b8e" (UID: "6e3fc4ac-760e-444b-a9eb-967f4d443b8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:01.446268 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.446242    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-kube-api-access-d4287\" (UniqueName: \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-kube-api-access-d4287\") pod \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") "
Nov 03 20:02:01.446399 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.446342    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-token\" (UniqueName: \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-token\") pod \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") "
Nov 03 20:02:01.446470 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.446411    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-certs\") pod \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\" (UID: \"6e3fc4ac-760e-444b-a9eb-967f4d443b8e\") "
Nov 03 20:02:01.446772 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.446746    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-certs" pod "6e3fc4ac-760e-444b-a9eb-967f4d443b8e" (UID: "6e3fc4ac-760e-444b-a9eb-967f4d443b8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:01.447629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.447603    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:01.447790 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.447633    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:01.447790 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.447649    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:01.447790 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.447661    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:01.447790 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.447674    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:01.448087 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.448063    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-podinfo" pod "6e3fc4ac-760e-444b-a9eb-967f4d443b8e" (UID: "6e3fc4ac-760e-444b-a9eb-967f4d443b8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:01.448228 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.448168    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-envoy" pod "6e3fc4ac-760e-444b-a9eb-967f4d443b8e" (UID: "6e3fc4ac-760e-444b-a9eb-967f4d443b8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:01.448949 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.448928    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-kube-api-access-d4287" pod "6e3fc4ac-760e-444b-a9eb-967f4d443b8e" (UID: "6e3fc4ac-760e-444b-a9eb-967f4d443b8e"). InnerVolumeSpecName "kube-api-access-d4287". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:01.449406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.449389    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-token" pod "6e3fc4ac-760e-444b-a9eb-967f4d443b8e" (UID: "6e3fc4ac-760e-444b-a9eb-967f4d443b8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:01.560198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.553061    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:01.560198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.553106    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:01.560198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.553123    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4287\" (UniqueName: \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-kube-api-access-d4287\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:01.560198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:01.553141    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3fc4ac-760e-444b-a9eb-967f4d443b8e-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:02.014717 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:02.014680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:02.326438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:02.326353    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"
Nov 03 20:02:02.326438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:02.326379    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd" event={"ID":"6e3fc4ac-760e-444b-a9eb-967f4d443b8e","Type":"ContainerDied","Data":"f010dc96bd081d990bd8fbb2725ae8e515c1e4181ea2c1a77714ff69eb6290aa"}
Nov 03 20:02:02.326942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:02.326447    4531 scope.go:122] "RemoveContainer" containerID="ac9467ce31127da3b59ab80b4abd00f54c3c38e37a395bc35d8f752eab378170"
Nov 03 20:02:02.357902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:02.357866    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"]
Nov 03 20:02:02.367297 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:02.367259    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-g74zd"]
Nov 03 20:02:02.864380 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:02.864345    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6551edf5-a4a6-4b4d-bb17-1b32b335c58e" path="/var/lib/kubelet/pods/6551edf5-a4a6-4b4d-bb17-1b32b335c58e/volumes"
Nov 03 20:02:02.865005 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:02.864989    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" path="/var/lib/kubelet/pods/6e3fc4ac-760e-444b-a9eb-967f4d443b8e/volumes"
Nov 03 20:02:02.865477 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:02.865463    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db439ef2-1c8e-4969-8c07-78c7ec87c3d9" path="/var/lib/kubelet/pods/db439ef2-1c8e-4969-8c07-78c7ec87c3d9/volumes"
Nov 03 20:02:02.982199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:02.977269    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:03.863341 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:03.863294    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:03.999369 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:03.999328    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:04.132616 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.132569    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:04.881830 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.881797    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 20:02:04.952300 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.943443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.972453    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-data\") pod \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") "
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.972782    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-credential-socket\") pod \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") "
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.972939    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-certs\") pod \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") "
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.972989    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-kube-api-access-46k5r\" (UniqueName: \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-kube-api-access-46k5r\") pod \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") "
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.973057    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istiod-ca-cert\") pod \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") "
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.974360    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-envoy\") pod \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") "
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.975055    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-podinfo\") pod \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") "
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.975297    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-socket\") pod \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") "
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.975458    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-token\" (UniqueName: \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-token\") pod \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\" (UID: \"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9\") "
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.979386    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-certs" pod "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" (UID: "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.979688    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-data" pod "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" (UID: "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.980519    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-credential-socket" pod "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" (UID: "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.980747    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-socket" pod "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" (UID: "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:04.986250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:04.981279    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istiod-ca-cert" pod "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" (UID: "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:05.074564 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.074523    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-podinfo" pod "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" (UID: "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:05.074724 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.074651    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-kube-api-access-46k5r" pod "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" (UID: "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9"). InnerVolumeSpecName "kube-api-access-46k5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:05.077805 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.077772    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-envoy" pod "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" (UID: "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.080416 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.080382    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.080416 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.080416    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.080617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.080431    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46k5r\" (UniqueName: \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-kube-api-access-46k5r\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.080617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.080446    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.080617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.080457    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.080617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.080469    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.080617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.080481    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.080617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.080492    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.080617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.080504    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-token" pod "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" (UID: "8e7ebaff-5e88-4e7e-a0d0-e446144c29f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:05.182925 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.182875    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.301185 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.301137    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 20:02:05.350032 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.344626    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" podUID="eede0436-4013-49f2-bbf6-00dade948787" containerName="main" containerID="cri-o://07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644" gracePeriod=45
Nov 03 20:02:05.366555 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.366510    4531 generic.go:356] "Generic (PLEG): container finished" podID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerID="73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434" exitCode=0
Nov 03 20:02:05.366780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.366570    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"
Nov 03 20:02:05.366780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.366605    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" event={"ID":"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9","Type":"ContainerDied","Data":"73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434"}
Nov 03 20:02:05.366780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.366669    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829" event={"ID":"8e7ebaff-5e88-4e7e-a0d0-e446144c29f9","Type":"ContainerDied","Data":"7e30ae2778a9b4e4848e11a425db299e960d10cc0516012e99aff6b9f8c353f0"}
Nov 03 20:02:05.366780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.366704    4531 scope.go:122] "RemoveContainer" containerID="73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434"
Nov 03 20:02:05.369050 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.369017    4531 generic.go:356] "Generic (PLEG): container finished" podID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerID="cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b" exitCode=0
Nov 03 20:02:05.369288 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.369249    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"
Nov 03 20:02:05.369400 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.369296    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" event={"ID":"30571081-5efa-4101-9cb2-06cd0be6d07c","Type":"ContainerDied","Data":"cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b"}
Nov 03 20:02:05.369400 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.369338    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm" event={"ID":"30571081-5efa-4101-9cb2-06cd0be6d07c","Type":"ContainerDied","Data":"d1824c74d63e4739c66852e54e34439184ab628513823506a926a14f95f71cff"}
Nov 03 20:02:05.385814 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.385780    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-token\" (UniqueName: \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-token\") pod \"30571081-5efa-4101-9cb2-06cd0be6d07c\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") "
Nov 03 20:02:05.385995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.385839    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-podinfo\") pod \"30571081-5efa-4101-9cb2-06cd0be6d07c\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") "
Nov 03 20:02:05.386077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.386016    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-kube-api-access-dbqjk\" (UniqueName: \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-kube-api-access-dbqjk\") pod \"30571081-5efa-4101-9cb2-06cd0be6d07c\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") "
Nov 03 20:02:05.386475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.386245    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-envoy\") pod \"30571081-5efa-4101-9cb2-06cd0be6d07c\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") "
Nov 03 20:02:05.386475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.386280    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-credential-socket\") pod \"30571081-5efa-4101-9cb2-06cd0be6d07c\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") "
Nov 03 20:02:05.386475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.386332    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-certs\") pod \"30571081-5efa-4101-9cb2-06cd0be6d07c\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") "
Nov 03 20:02:05.386475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.386473    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30571081-5efa-4101-9cb2-06cd0be6d07c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30571081-5efa-4101-9cb2-06cd0be6d07c-istiod-ca-cert\") pod \"30571081-5efa-4101-9cb2-06cd0be6d07c\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") "
Nov 03 20:02:05.386883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.386577    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-data\") pod \"30571081-5efa-4101-9cb2-06cd0be6d07c\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") "
Nov 03 20:02:05.386883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.386696    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-socket\") pod \"30571081-5efa-4101-9cb2-06cd0be6d07c\" (UID: \"30571081-5efa-4101-9cb2-06cd0be6d07c\") "
Nov 03 20:02:05.388823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.387967    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-socket" pod "30571081-5efa-4101-9cb2-06cd0be6d07c" (UID: "30571081-5efa-4101-9cb2-06cd0be6d07c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.388823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.388185    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-credential-socket" pod "30571081-5efa-4101-9cb2-06cd0be6d07c" (UID: "30571081-5efa-4101-9cb2-06cd0be6d07c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.388823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.388364    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-certs" pod "30571081-5efa-4101-9cb2-06cd0be6d07c" (UID: "30571081-5efa-4101-9cb2-06cd0be6d07c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.388823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.388723    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30571081-5efa-4101-9cb2-06cd0be6d07c-istiod-ca-cert" pod "30571081-5efa-4101-9cb2-06cd0be6d07c" (UID: "30571081-5efa-4101-9cb2-06cd0be6d07c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:05.389235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.389034    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-data" pod "30571081-5efa-4101-9cb2-06cd0be6d07c" (UID: "30571081-5efa-4101-9cb2-06cd0be6d07c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.390715 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.390690    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-envoy" pod "30571081-5efa-4101-9cb2-06cd0be6d07c" (UID: "30571081-5efa-4101-9cb2-06cd0be6d07c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.390840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.390733    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-podinfo" pod "30571081-5efa-4101-9cb2-06cd0be6d07c" (UID: "30571081-5efa-4101-9cb2-06cd0be6d07c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:05.391249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.391147    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-token" pod "30571081-5efa-4101-9cb2-06cd0be6d07c" (UID: "30571081-5efa-4101-9cb2-06cd0be6d07c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:05.392158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.392131    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-kube-api-access-dbqjk" pod "30571081-5efa-4101-9cb2-06cd0be6d07c" (UID: "30571081-5efa-4101-9cb2-06cd0be6d07c"). InnerVolumeSpecName "kube-api-access-dbqjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:05.402900 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.402865    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" podUID="eede0436-4013-49f2-bbf6-00dade948787" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/ready\": dial tcp 10.133.0.197:8000: connect: connection refused"
Nov 03 20:02:05.410938 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.410913    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"]
Nov 03 20:02:05.416306 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.416286    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-hs829"]
Nov 03 20:02:05.484426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.484396    4531 scope.go:122] "RemoveContainer" containerID="73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434"
Nov 03 20:02:05.487836 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:05.487797    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434\": container with ID starting with 73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434 not found: ID does not exist" containerID="73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434"
Nov 03 20:02:05.487941 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.487843    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434"} err="failed to get container status \"73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434\": rpc error: code = NotFound desc = could not find container \"73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434\": container with ID starting with 73dd6734246a827e3fe058140110915b34f3f7c82321a0f392946b6f2c6d1434 not found: ID does not exist"
Nov 03 20:02:05.487941 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.487867    4531 scope.go:122] "RemoveContainer" containerID="cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b"
Nov 03 20:02:05.490150 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.490121    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30571081-5efa-4101-9cb2-06cd0be6d07c-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.490283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.490156    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.490283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.490189    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.490283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.490202    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.490283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.490215    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.490283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.490230    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbqjk\" (UniqueName: \"kubernetes.io/projected/30571081-5efa-4101-9cb2-06cd0be6d07c-kube-api-access-dbqjk\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.490283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.490247    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.490283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.490261    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.490283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.490275    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30571081-5efa-4101-9cb2-06cd0be6d07c-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.509887 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.509861    4531 scope.go:122] "RemoveContainer" containerID="cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b"
Nov 03 20:02:05.510431 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:05.510396    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b\": container with ID starting with cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b not found: ID does not exist" containerID="cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b"
Nov 03 20:02:05.510554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.510442    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b"} err="failed to get container status \"cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b\": rpc error: code = NotFound desc = could not find container \"cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b\": container with ID starting with cac7b161778d017ea73ba9983a49c3b03730f788949ec0bb509bc7011ab67e1b not found: ID does not exist"
Nov 03 20:02:05.631868 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.631819    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:05.723634 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.723569    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"]
Nov 03 20:02:05.732488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.732323    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-vn9nm"]
Nov 03 20:02:05.760825 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.760784    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:05.765457 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.765425    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 20:02:05.795589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.795551    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-tmp-dir\") pod \"eede0436-4013-49f2-bbf6-00dade948787\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") "
Nov 03 20:02:05.795771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.795689    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eede0436-4013-49f2-bbf6-00dade948787-tls-certs\" (UniqueName: \"kubernetes.io/secret/eede0436-4013-49f2-bbf6-00dade948787-tls-certs\") pod \"eede0436-4013-49f2-bbf6-00dade948787\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") "
Nov 03 20:02:05.795771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.795733    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-model-cache\") pod \"eede0436-4013-49f2-bbf6-00dade948787\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") "
Nov 03 20:02:05.795914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.795883    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-home\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-home\") pod \"eede0436-4013-49f2-bbf6-00dade948787\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") "
Nov 03 20:02:05.796301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.796061    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eede0436-4013-49f2-bbf6-00dade948787-kube-api-access-t5czz\" (UniqueName: \"kubernetes.io/projected/eede0436-4013-49f2-bbf6-00dade948787-kube-api-access-t5czz\") pod \"eede0436-4013-49f2-bbf6-00dade948787\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") "
Nov 03 20:02:05.796301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.795878    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-tmp-dir" pod "eede0436-4013-49f2-bbf6-00dade948787" (UID: "eede0436-4013-49f2-bbf6-00dade948787"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.796301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.796068    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-model-cache" pod "eede0436-4013-49f2-bbf6-00dade948787" (UID: "eede0436-4013-49f2-bbf6-00dade948787"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.796301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.796103    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-dshm\") pod \"eede0436-4013-49f2-bbf6-00dade948787\" (UID: \"eede0436-4013-49f2-bbf6-00dade948787\") "
Nov 03 20:02:05.797326 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.796641    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-home" pod "eede0436-4013-49f2-bbf6-00dade948787" (UID: "eede0436-4013-49f2-bbf6-00dade948787"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.797326 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.797260    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.797326 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.797286    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.797326 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.797301    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.801223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.799054    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-dshm" pod "eede0436-4013-49f2-bbf6-00dade948787" (UID: "eede0436-4013-49f2-bbf6-00dade948787"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.801223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.799062    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eede0436-4013-49f2-bbf6-00dade948787-tls-certs" pod "eede0436-4013-49f2-bbf6-00dade948787" (UID: "eede0436-4013-49f2-bbf6-00dade948787"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:05.801223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.799061    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eede0436-4013-49f2-bbf6-00dade948787-kube-api-access-t5czz" pod "eede0436-4013-49f2-bbf6-00dade948787" (UID: "eede0436-4013-49f2-bbf6-00dade948787"). InnerVolumeSpecName "kube-api-access-t5czz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:05.899743 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.899709    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5czz\" (UniqueName: \"kubernetes.io/projected/eede0436-4013-49f2-bbf6-00dade948787-kube-api-access-t5czz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.899743 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.899739    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eede0436-4013-49f2-bbf6-00dade948787-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.899743 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:05.899749    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eede0436-4013-49f2-bbf6-00dade948787-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:06.077200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.069952    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:06.283457 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.283363    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp" podUID="30c45340-7f63-4f26-8eeb-9efc99a62060" containerName="payload-pre-processing" containerID="cri-o://c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69" gracePeriod=25
Nov 03 20:02:06.287935 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.287899    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp" podUID="318e30af-1cb1-4dff-8f0a-bdd066e0078b" containerName="payload-processing" containerID="cri-o://7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3" gracePeriod=25
Nov 03 20:02:06.381050 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.381014    4531 generic.go:356] "Generic (PLEG): container finished" podID="eede0436-4013-49f2-bbf6-00dade948787" containerID="07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644" exitCode=0
Nov 03 20:02:06.381260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.381067    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"
Nov 03 20:02:06.381260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.381097    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" event={"ID":"eede0436-4013-49f2-bbf6-00dade948787","Type":"ContainerDied","Data":"07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644"}
Nov 03 20:02:06.381260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.381139    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b" event={"ID":"eede0436-4013-49f2-bbf6-00dade948787","Type":"ContainerDied","Data":"7ed800307495b3ff5afece077a410a073bfe9231b5b3707f1735a43b39233e0c"}
Nov 03 20:02:06.381443 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.381256    4531 scope.go:122] "RemoveContainer" containerID="07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644"
Nov 03 20:02:06.395777 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.395755    4531 scope.go:122] "RemoveContainer" containerID="07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644"
Nov 03 20:02:06.396147 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:06.396124    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644\": container with ID starting with 07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644 not found: ID does not exist" containerID="07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644"
Nov 03 20:02:06.396291 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.396155    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644"} err="failed to get container status \"07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644\": rpc error: code = NotFound desc = could not find container \"07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644\": container with ID starting with 07d05976fdfc9fc3f7e8ce04ae67756c2fdc98496a8875e964f6c4ddc975b644 not found: ID does not exist"
Nov 03 20:02:06.410394 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.410362    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"]
Nov 03 20:02:06.415348 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.415315    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a99fad/test-model-a99fad-kserve-7f8946b455-hmv9b"]
Nov 03 20:02:06.730187 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.730139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:06.874453 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.874416    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" path="/var/lib/kubelet/pods/30571081-5efa-4101-9cb2-06cd0be6d07c/volumes"
Nov 03 20:02:06.875242 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.875224    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" path="/var/lib/kubelet/pods/8e7ebaff-5e88-4e7e-a0d0-e446144c29f9/volumes"
Nov 03 20:02:06.875835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:06.875819    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eede0436-4013-49f2-bbf6-00dade948787" path="/var/lib/kubelet/pods/eede0436-4013-49f2-bbf6-00dade948787/volumes"
Nov 03 20:02:07.105252 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.079839    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:07.361250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.354118    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"]
Nov 03 20:02:07.384931 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.371598    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerName="istio-proxy"
Nov 03 20:02:07.411760 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.411720    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerName="istio-proxy"
Nov 03 20:02:07.417201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.412036    4531 cpu_manager.go:414] "containerMap: removing container" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerName="istio-proxy"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.495780    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerName="istio-proxy"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.495866    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerName="istio-proxy"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.495891    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerName="istio-proxy"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.495907    4531 cpu_manager.go:414] "containerMap: removing container" podUID="db439ef2-1c8e-4969-8c07-78c7ec87c3d9" containerName="payload-processing"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.495915    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db439ef2-1c8e-4969-8c07-78c7ec87c3d9" containerName="payload-processing"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.495943    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6551edf5-a4a6-4b4d-bb17-1b32b335c58e" containerName="payload-pre-processing"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.495970    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6551edf5-a4a6-4b4d-bb17-1b32b335c58e" containerName="payload-pre-processing"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.496068    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.496078    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.496100    4531 cpu_manager.go:414] "containerMap: removing container" podUID="baa40e8f-14ac-4070-8d1d-1276da8d481a" containerName="revoke-keys"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.496121    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa40e8f-14ac-4070-8d1d-1276da8d481a" containerName="revoke-keys"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.496145    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cd542f86-cf28-4141-b7f7-b0661226c455" containerName="main"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.496152    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd542f86-cf28-4141-b7f7-b0661226c455" containerName="main"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.496201    4531 cpu_manager.go:414] "containerMap: removing container" podUID="eede0436-4013-49f2-bbf6-00dade948787" containerName="main"
Nov 03 20:02:07.496196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.496209    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eede0436-4013-49f2-bbf6-00dade948787" containerName="main"
Nov 03 20:02:07.590385 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.468302    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:07.612282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.612193    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7ebaff-5e88-4e7e-a0d0-e446144c29f9" containerName="istio-proxy"
Nov 03 20:02:07.612538 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.612522    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="30571081-5efa-4101-9cb2-06cd0be6d07c" containerName="istio-proxy"
Nov 03 20:02:07.612644 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.612633    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3fc4ac-760e-444b-a9eb-967f4d443b8e" containerName="istio-proxy"
Nov 03 20:02:07.612733 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.612723    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="db439ef2-1c8e-4969-8c07-78c7ec87c3d9" containerName="payload-processing"
Nov 03 20:02:07.612830 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.612820    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6551edf5-a4a6-4b4d-bb17-1b32b335c58e" containerName="payload-pre-processing"
Nov 03 20:02:07.612942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.612930    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5db9016-7dd6-4881-8e89-2bb9751a23c1" containerName="istio-proxy"
Nov 03 20:02:07.613033 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.613023    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa40e8f-14ac-4070-8d1d-1276da8d481a" containerName="revoke-keys"
Nov 03 20:02:07.613127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.613116    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd542f86-cf28-4141-b7f7-b0661226c455" containerName="main"
Nov 03 20:02:07.613318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.613302    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="eede0436-4013-49f2-bbf6-00dade948787" containerName="main"
Nov 03 20:02:07.703698 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.703662    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:02:07.751448 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.750273    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"]
Nov 03 20:02:07.764971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.758823    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:07.764971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.762684    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-3ff278\""
Nov 03 20:02:07.788195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.775045    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"]
Nov 03 20:02:07.810194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.793801    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:02:07.810194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.797489    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-3ff278-dockercfg-5qmrt\""
Nov 03 20:02:07.841196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.836609    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"]
Nov 03 20:02:07.940978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.940943    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7c7784-0528-47bd-b6a8-b42778e96d58-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd\" (UID: \"0b7c7784-0528-47bd-b6a8-b42778e96d58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:02:07.941256 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:07.941234    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r74j\" (UniqueName: \"kubernetes.io/projected/0b7c7784-0528-47bd-b6a8-b42778e96d58-kube-api-access-2r74j\") pod \"payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd\" (UID: \"0b7c7784-0528-47bd-b6a8-b42778e96d58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:02:08.072442 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.067680    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7987b86a-8744-4e09-a949-c23671975702-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h\" (UID: \"7987b86a-8744-4e09-a949-c23671975702\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:02:08.075422 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.075388    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89x55\" (UniqueName: \"kubernetes.io/projected/7987b86a-8744-4e09-a949-c23671975702-kube-api-access-89x55\") pod \"payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h\" (UID: \"7987b86a-8744-4e09-a949-c23671975702\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:02:08.075768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.075749    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7c7784-0528-47bd-b6a8-b42778e96d58-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd\" (UID: \"0b7c7784-0528-47bd-b6a8-b42778e96d58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:02:08.075969 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.075951    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r74j\" (UniqueName: \"kubernetes.io/projected/0b7c7784-0528-47bd-b6a8-b42778e96d58-kube-api-access-2r74j\") pod \"payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd\" (UID: \"0b7c7784-0528-47bd-b6a8-b42778e96d58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:02:08.077640 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.077607    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7c7784-0528-47bd-b6a8-b42778e96d58-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd\" (UID: \"0b7c7784-0528-47bd-b6a8-b42778e96d58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:02:08.085695 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.085667    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r74j\" (UniqueName: \"kubernetes.io/projected/0b7c7784-0528-47bd-b6a8-b42778e96d58-kube-api-access-2r74j\") pod \"payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd\" (UID: \"0b7c7784-0528-47bd-b6a8-b42778e96d58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:02:08.178362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.178261    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7987b86a-8744-4e09-a949-c23671975702-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h\" (UID: \"7987b86a-8744-4e09-a949-c23671975702\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:02:08.178874 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.178481    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89x55\" (UniqueName: \"kubernetes.io/projected/7987b86a-8744-4e09-a949-c23671975702-kube-api-access-89x55\") pod \"payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h\" (UID: \"7987b86a-8744-4e09-a949-c23671975702\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:02:08.179162 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.179138    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7987b86a-8744-4e09-a949-c23671975702-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h\" (UID: \"7987b86a-8744-4e09-a949-c23671975702\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:02:08.186149 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.186104    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89x55\" (UniqueName: \"kubernetes.io/projected/7987b86a-8744-4e09-a949-c23671975702-kube-api-access-89x55\") pod \"payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h\" (UID: \"7987b86a-8744-4e09-a949-c23671975702\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:02:08.261599 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.261564    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:02:08.268283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.268252    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:02:08.338605 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.338565    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:08.532336 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.532240    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp" podUID="30c45340-7f63-4f26-8eeb-9efc99a62060" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:08.561091 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:02:08.561059    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7c7784_0528_47bd_b6a8_b42778e96d58.slice/crio-e9437cf6b8ac103716ea5ea87fe94fb2a973c6966897518949dacf466e3c99f8 WatchSource:0}: Error finding container e9437cf6b8ac103716ea5ea87fe94fb2a973c6966897518949dacf466e3c99f8: Status 404 returned error can't find the container with id e9437cf6b8ac103716ea5ea87fe94fb2a973c6966897518949dacf466e3c99f8
Nov 03 20:02:08.561594 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.561565    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"]
Nov 03 20:02:08.577159 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.577132    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"]
Nov 03 20:02:08.578936 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:02:08.578908    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7987b86a_8744_4e09_a949_c23671975702.slice/crio-180828f720f68e57b1e295141f034eaf50590789e6ffea2c7f4b58536c039cd7 WatchSource:0}: Error finding container 180828f720f68e57b1e295141f034eaf50590789e6ffea2c7f4b58536c039cd7: Status 404 returned error can't find the container with id 180828f720f68e57b1e295141f034eaf50590789e6ffea2c7f4b58536c039cd7
Nov 03 20:02:08.795330 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.793786    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:08.920161 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.920126    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h" event={"ID":"7987b86a-8744-4e09-a949-c23671975702","Type":"ContainerStarted","Data":"3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f"}
Nov 03 20:02:08.920361 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.920203    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h" event={"ID":"7987b86a-8744-4e09-a949-c23671975702","Type":"ContainerStarted","Data":"180828f720f68e57b1e295141f034eaf50590789e6ffea2c7f4b58536c039cd7"}
Nov 03 20:02:08.920361 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.920228    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:02:08.920361 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.920244    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd" event={"ID":"0b7c7784-0528-47bd-b6a8-b42778e96d58","Type":"ContainerStarted","Data":"1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6"}
Nov 03 20:02:08.920361 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.920261    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:02:08.920361 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.920278    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd" event={"ID":"0b7c7784-0528-47bd-b6a8-b42778e96d58","Type":"ContainerStarted","Data":"e9437cf6b8ac103716ea5ea87fe94fb2a973c6966897518949dacf466e3c99f8"}
Nov 03 20:02:08.929414 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.929361    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h" podStartSLOduration=1.929347283 podStartE2EDuration="1.929347283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:08.924462346 +0000 UTC m=+1842.194585641" watchObservedRunningTime="2026-11-03 20:02:08.929347283 +0000 UTC m=+1842.199470595"
Nov 03 20:02:08.941634 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:08.941567    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd" podStartSLOduration=1.941549459 podStartE2EDuration="1.941549459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:08.937383414 +0000 UTC m=+1842.207506708" watchObservedRunningTime="2026-11-03 20:02:08.941549459 +0000 UTC m=+1842.211672755"
Nov 03 20:02:09.943065 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:09.942944    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp" podUID="318e30af-1cb1-4dff-8f0a-bdd066e0078b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:09.944203 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:09.943522    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:10.869739 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:10.869697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:11.533267 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:11.533217    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:12.239201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:12.238412    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:12.944199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:12.937321    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:13.674021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:13.672697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:14.227310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:14.227259    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:14.505729 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:14.505623    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:14.680206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:14.673136    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:15.448901 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:15.448848    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:16.142203 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:16.140634    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:16.377986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:16.377950    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:16.877323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:16.877291    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:17.012387 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:17.012335    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:17.844195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:17.835133    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:18.449213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:18.448051    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:18.534775 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:18.534728    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp" podUID="30c45340-7f63-4f26-8eeb-9efc99a62060" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:19.305240 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:19.305161    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:19.916595 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:19.916552    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp" podUID="318e30af-1cb1-4dff-8f0a-bdd066e0078b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:20.037720 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:20.037689    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:02:20.037924 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:20.037780    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:02:20.255198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:20.237260    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:20.580687 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:20.580589    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:02:20.617827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:20.617792    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"]
Nov 03 20:02:20.674761 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:20.674721    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"]
Nov 03 20:02:20.675728 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:20.675683    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh" podUID="0eb4dd53-79c8-4365-96c7-0a7644c56382" containerName="maas-api" containerID="cri-o://6473d02b53b2dd8a8ef3e926e6cb0110df4d44612b81fec6f4d3afcb6ee2b737" gracePeriod=30
Nov 03 20:02:20.708254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:20.708193    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:20.844245 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:20.844139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:21.003471 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.003423    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:21.155205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.143765    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:21.219198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.218252    4531 generic.go:356] "Generic (PLEG): container finished" podID="0eb4dd53-79c8-4365-96c7-0a7644c56382" containerID="6473d02b53b2dd8a8ef3e926e6cb0110df4d44612b81fec6f4d3afcb6ee2b737" exitCode=0
Nov 03 20:02:21.219198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.218358    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh" event={"ID":"0eb4dd53-79c8-4365-96c7-0a7644c56382","Type":"ContainerDied","Data":"6473d02b53b2dd8a8ef3e926e6cb0110df4d44612b81fec6f4d3afcb6ee2b737"}
Nov 03 20:02:21.425251 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.419847    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"]
Nov 03 20:02:21.459197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.446225    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:21.595226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.595120    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:21.784881 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.784851    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 20:02:21.823979 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.823939    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb4dd53-79c8-4365-96c7-0a7644c56382-kube-api-access-g24r8\" (UniqueName: \"kubernetes.io/projected/0eb4dd53-79c8-4365-96c7-0a7644c56382-kube-api-access-g24r8\") pod \"0eb4dd53-79c8-4365-96c7-0a7644c56382\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") "
Nov 03 20:02:21.824236 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.824087    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-metrics-certs\") pod \"0eb4dd53-79c8-4365-96c7-0a7644c56382\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") "
Nov 03 20:02:21.824421 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.824393    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb4dd53-79c8-4365-96c7-0a7644c56382-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb4dd53-79c8-4365-96c7-0a7644c56382-openshift-service-ca\") pod \"0eb4dd53-79c8-4365-96c7-0a7644c56382\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") "
Nov 03 20:02:21.824521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.824494    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-maas-api-tls\") pod \"0eb4dd53-79c8-4365-96c7-0a7644c56382\" (UID: \"0eb4dd53-79c8-4365-96c7-0a7644c56382\") "
Nov 03 20:02:21.824840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.824811    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb4dd53-79c8-4365-96c7-0a7644c56382-openshift-service-ca" pod "0eb4dd53-79c8-4365-96c7-0a7644c56382" (UID: "0eb4dd53-79c8-4365-96c7-0a7644c56382"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:21.825518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.825488    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb4dd53-79c8-4365-96c7-0a7644c56382-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:21.826780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.826747    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb4dd53-79c8-4365-96c7-0a7644c56382-kube-api-access-g24r8" pod "0eb4dd53-79c8-4365-96c7-0a7644c56382" (UID: "0eb4dd53-79c8-4365-96c7-0a7644c56382"). InnerVolumeSpecName "kube-api-access-g24r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:21.827578 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.827524    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-maas-api-tls" pod "0eb4dd53-79c8-4365-96c7-0a7644c56382" (UID: "0eb4dd53-79c8-4365-96c7-0a7644c56382"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:21.829744 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.829663    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-metrics-certs" pod "0eb4dd53-79c8-4365-96c7-0a7644c56382" (UID: "0eb4dd53-79c8-4365-96c7-0a7644c56382"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:21.929283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.929219    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:21.929503 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.929290    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g24r8\" (UniqueName: \"kubernetes.io/projected/0eb4dd53-79c8-4365-96c7-0a7644c56382-kube-api-access-g24r8\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:21.929503 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:21.929309    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb4dd53-79c8-4365-96c7-0a7644c56382-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:22.229623 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:22.229590    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"
Nov 03 20:02:22.230095 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:22.229619    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh" event={"ID":"0eb4dd53-79c8-4365-96c7-0a7644c56382","Type":"ContainerDied","Data":"304e6f43b883501952e9889843cd82eef168e04699cb9f0a644cb6b34b04077d"}
Nov 03 20:02:22.230095 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:22.229697    4531 scope.go:122] "RemoveContainer" containerID="6473d02b53b2dd8a8ef3e926e6cb0110df4d44612b81fec6f4d3afcb6ee2b737"
Nov 03 20:02:22.255559 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:22.255523    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"]
Nov 03 20:02:22.260867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:22.260836    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ffe132-577d4b985f-6sndh"]
Nov 03 20:02:22.347199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:22.347013    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:22.866138 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:22.866102    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb4dd53-79c8-4365-96c7-0a7644c56382" path="/var/lib/kubelet/pods/0eb4dd53-79c8-4365-96c7-0a7644c56382/volumes"
Nov 03 20:02:23.051568 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:23.047804    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:23.844200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:23.837889    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:24.651197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:24.649985    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:25.401407 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:25.392817    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:26.061199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.056837    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:26.665738 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.665702    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 20:02:26.671825 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.671788    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 20:02:26.674354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.674322    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c45340-7f63-4f26-8eeb-9efc99a62060-kube-api-access-wx554\" (UniqueName: \"kubernetes.io/projected/30c45340-7f63-4f26-8eeb-9efc99a62060-kube-api-access-wx554\") pod \"30c45340-7f63-4f26-8eeb-9efc99a62060\" (UID: \"30c45340-7f63-4f26-8eeb-9efc99a62060\") "
Nov 03 20:02:26.674891 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.674862    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c45340-7f63-4f26-8eeb-9efc99a62060-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c45340-7f63-4f26-8eeb-9efc99a62060-plugins-config-volume\") pod \"30c45340-7f63-4f26-8eeb-9efc99a62060\" (UID: \"30c45340-7f63-4f26-8eeb-9efc99a62060\") "
Nov 03 20:02:26.676575 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.676543    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c45340-7f63-4f26-8eeb-9efc99a62060-plugins-config-volume" pod "30c45340-7f63-4f26-8eeb-9efc99a62060" (UID: "30c45340-7f63-4f26-8eeb-9efc99a62060"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:26.678752 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.678726    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c45340-7f63-4f26-8eeb-9efc99a62060-kube-api-access-wx554" pod "30c45340-7f63-4f26-8eeb-9efc99a62060" (UID: "30c45340-7f63-4f26-8eeb-9efc99a62060"). InnerVolumeSpecName "kube-api-access-wx554". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:26.742490 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.742450    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:26.776980 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.776932    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318e30af-1cb1-4dff-8f0a-bdd066e0078b-kube-api-access-8blgd\" (UniqueName: \"kubernetes.io/projected/318e30af-1cb1-4dff-8f0a-bdd066e0078b-kube-api-access-8blgd\") pod \"318e30af-1cb1-4dff-8f0a-bdd066e0078b\" (UID: \"318e30af-1cb1-4dff-8f0a-bdd066e0078b\") "
Nov 03 20:02:26.777251 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.777189    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318e30af-1cb1-4dff-8f0a-bdd066e0078b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318e30af-1cb1-4dff-8f0a-bdd066e0078b-plugins-config-volume\") pod \"318e30af-1cb1-4dff-8f0a-bdd066e0078b\" (UID: \"318e30af-1cb1-4dff-8f0a-bdd066e0078b\") "
Nov 03 20:02:26.777606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.777567    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318e30af-1cb1-4dff-8f0a-bdd066e0078b-plugins-config-volume" pod "318e30af-1cb1-4dff-8f0a-bdd066e0078b" (UID: "318e30af-1cb1-4dff-8f0a-bdd066e0078b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:26.778331 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.778307    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c45340-7f63-4f26-8eeb-9efc99a62060-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:26.778389 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.778340    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318e30af-1cb1-4dff-8f0a-bdd066e0078b-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:26.778389 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.778356    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx554\" (UniqueName: \"kubernetes.io/projected/30c45340-7f63-4f26-8eeb-9efc99a62060-kube-api-access-wx554\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:26.779165 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.779141    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318e30af-1cb1-4dff-8f0a-bdd066e0078b-kube-api-access-8blgd" pod "318e30af-1cb1-4dff-8f0a-bdd066e0078b" (UID: "318e30af-1cb1-4dff-8f0a-bdd066e0078b"). InnerVolumeSpecName "kube-api-access-8blgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:26.880838 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.880803    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8blgd\" (UniqueName: \"kubernetes.io/projected/318e30af-1cb1-4dff-8f0a-bdd066e0078b-kube-api-access-8blgd\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:26.892914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:26.892848    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:27.117198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.110221    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:27.420785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.390724    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:27.503939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.491648    4531 generic.go:356] "Generic (PLEG): container finished" podID="30c45340-7f63-4f26-8eeb-9efc99a62060" containerID="c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69" exitCode=0
Nov 03 20:02:27.503939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.492124    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp" event={"ID":"30c45340-7f63-4f26-8eeb-9efc99a62060","Type":"ContainerDied","Data":"c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69"}
Nov 03 20:02:27.503939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.492297    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp" event={"ID":"30c45340-7f63-4f26-8eeb-9efc99a62060","Type":"ContainerDied","Data":"ffb69eb05d7fe255e84993de0512f0768b7b49efaddf2c8e12275b40ca4591b7"}
Nov 03 20:02:27.503939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.492370    4531 scope.go:122] "RemoveContainer" containerID="c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69"
Nov 03 20:02:27.503939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.492677    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"
Nov 03 20:02:27.557634 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.557593    4531 generic.go:356] "Generic (PLEG): container finished" podID="318e30af-1cb1-4dff-8f0a-bdd066e0078b" containerID="7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3" exitCode=0
Nov 03 20:02:27.568009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.558313    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"
Nov 03 20:02:27.572116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.558166    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp" event={"ID":"318e30af-1cb1-4dff-8f0a-bdd066e0078b","Type":"ContainerDied","Data":"7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3"}
Nov 03 20:02:27.572116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.570039    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"]
Nov 03 20:02:27.572116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.570079    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp" event={"ID":"318e30af-1cb1-4dff-8f0a-bdd066e0078b","Type":"ContainerDied","Data":"e96617d7b6e21c174b3e4d07b528eec1eeefec4004a4da07f2958a9de6c2955d"}
Nov 03 20:02:27.572849 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.572820    4531 scope.go:122] "RemoveContainer" containerID="c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69"
Nov 03 20:02:27.581212 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:27.581149    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69\": container with ID starting with c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69 not found: ID does not exist" containerID="c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69"
Nov 03 20:02:27.581352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.581230    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69"} err="failed to get container status \"c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69\": rpc error: code = NotFound desc = could not find container \"c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69\": container with ID starting with c7be606a8b0f68116f032cb74daf4c72194b833f2b92d1aaf41ee2e76f821e69 not found: ID does not exist"
Nov 03 20:02:27.581352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.581292    4531 scope.go:122] "RemoveContainer" containerID="7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3"
Nov 03 20:02:27.612741 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.612703    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ffe132-c9cbc6c4c-br8pp"]
Nov 03 20:02:27.623709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.623676    4531 scope.go:122] "RemoveContainer" containerID="7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3"
Nov 03 20:02:27.624617 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:27.624577    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3\": container with ID starting with 7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3 not found: ID does not exist" containerID="7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3"
Nov 03 20:02:27.624772 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.624625    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3"} err="failed to get container status \"7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3\": rpc error: code = NotFound desc = could not find container \"7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3\": container with ID starting with 7a539f429be0b15b8fadb8d4093b74bc73153c52b2ce023977ffa9af437ec4a3 not found: ID does not exist"
Nov 03 20:02:27.626024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.625999    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"]
Nov 03 20:02:27.631036 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.631006    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ffe132-d646f478c-9zxxp"]
Nov 03 20:02:27.767808 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:27.767772    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:28.190150 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:28.190110    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:28.458851 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:28.458751    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:28.603195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:28.596014    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:28.699927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:28.699880    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"]
Nov 03 20:02:28.700225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:28.700150    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy" containerID="cri-o://b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7" gracePeriod=30
Nov 03 20:02:28.708768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:28.708737    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"]
Nov 03 20:02:28.709077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:28.709009    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy" containerID="cri-o://be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052" gracePeriod=30
Nov 03 20:02:28.785469 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:28.785416    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:29.079258 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.074386    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c45340-7f63-4f26-8eeb-9efc99a62060" path="/var/lib/kubelet/pods/30c45340-7f63-4f26-8eeb-9efc99a62060/volumes"
Nov 03 20:02:29.091672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.091326    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318e30af-1cb1-4dff-8f0a-bdd066e0078b" path="/var/lib/kubelet/pods/318e30af-1cb1-4dff-8f0a-bdd066e0078b/volumes"
Nov 03 20:02:29.250150 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.250113    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"]
Nov 03 20:02:29.251086 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.251066    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0eb4dd53-79c8-4365-96c7-0a7644c56382" containerName="maas-api"
Nov 03 20:02:29.251133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.251088    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb4dd53-79c8-4365-96c7-0a7644c56382" containerName="maas-api"
Nov 03 20:02:29.251133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.251103    4531 cpu_manager.go:414] "containerMap: removing container" podUID="30c45340-7f63-4f26-8eeb-9efc99a62060" containerName="payload-pre-processing"
Nov 03 20:02:29.251133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.251110    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c45340-7f63-4f26-8eeb-9efc99a62060" containerName="payload-pre-processing"
Nov 03 20:02:29.251133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.251129    4531 cpu_manager.go:414] "containerMap: removing container" podUID="318e30af-1cb1-4dff-8f0a-bdd066e0078b" containerName="payload-processing"
Nov 03 20:02:29.251347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.251134    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318e30af-1cb1-4dff-8f0a-bdd066e0078b" containerName="payload-processing"
Nov 03 20:02:29.251347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.251286    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="318e30af-1cb1-4dff-8f0a-bdd066e0078b" containerName="payload-processing"
Nov 03 20:02:29.251347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.251306    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb4dd53-79c8-4365-96c7-0a7644c56382" containerName="maas-api"
Nov 03 20:02:29.251347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.251315    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c45340-7f63-4f26-8eeb-9efc99a62060" containerName="payload-pre-processing"
Nov 03 20:02:29.257359 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.257332    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:29.260677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.260639    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"]
Nov 03 20:02:29.263136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.263109    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:02:29.263565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.263539    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:02:29.364487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.364399    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:29.364487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.364464    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbqz\" (UniqueName: \"kubernetes.io/projected/4cad6d22-639f-45c0-a03f-b7e04095b3ad-kube-api-access-gzbqz\") pod \"maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:29.364708 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.364622    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:29.466645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.466589    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:29.466821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.466714    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbqz\" (UniqueName: \"kubernetes.io/projected/4cad6d22-639f-45c0-a03f-b7e04095b3ad-kube-api-access-gzbqz\") pod \"maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:29.466907 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.466890    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:29.467635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.467608    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:29.469215 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.469192    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:29.476789 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.476754    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbqz\" (UniqueName: \"kubernetes.io/projected/4cad6d22-639f-45c0-a03f-b7e04095b3ad-kube-api-access-gzbqz\") pod \"maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:29.533280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.521219    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:29.585895 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.583737    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:29.955517 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.955437    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:02:29.955517 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.955501    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:02:29.978322 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:02:29.978289    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cad6d22_639f_45c0_a03f_b7e04095b3ad.slice/crio-05fead65f96ba947fcac7458c8ec15e655ab2e79f814d5dbcacab288309ab189 WatchSource:0}: Error finding container 05fead65f96ba947fcac7458c8ec15e655ab2e79f814d5dbcacab288309ab189: Status 404 returned error can't find the container with id 05fead65f96ba947fcac7458c8ec15e655ab2e79f814d5dbcacab288309ab189
Nov 03 20:02:29.979149 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:29.979122    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"]
Nov 03 20:02:30.390412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:30.390367    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:30.680562 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:30.680522    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj" event={"ID":"4cad6d22-639f-45c0-a03f-b7e04095b3ad","Type":"ContainerStarted","Data":"1e8c3ed107673db6d3fe42cc6c5cf800bb2f361e2c56b5a03fc339be76c23308"}
Nov 03 20:02:30.680762 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:30.680590    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj" event={"ID":"4cad6d22-639f-45c0-a03f-b7e04095b3ad","Type":"ContainerStarted","Data":"05fead65f96ba947fcac7458c8ec15e655ab2e79f814d5dbcacab288309ab189"}
Nov 03 20:02:30.700114 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:30.700052    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj" podStartSLOduration=1.7000354739999999 podStartE2EDuration="1.700035474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:30.695632565 +0000 UTC m=+1863.965755859" watchObservedRunningTime="2026-11-03 20:02:30.700035474 +0000 UTC m=+1863.970158785"
Nov 03 20:02:31.254252 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:31.234610    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:31.691336 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:31.691299    4531 generic.go:356] "Generic (PLEG): container finished" podID="4cad6d22-639f-45c0-a03f-b7e04095b3ad" containerID="1e8c3ed107673db6d3fe42cc6c5cf800bb2f361e2c56b5a03fc339be76c23308" exitCode=0
Nov 03 20:02:31.691553 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:31.691380    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj" event={"ID":"4cad6d22-639f-45c0-a03f-b7e04095b3ad","Type":"ContainerDied","Data":"1e8c3ed107673db6d3fe42cc6c5cf800bb2f361e2c56b5a03fc339be76c23308"}
Nov 03 20:02:31.881483 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:31.881444    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:32.088575 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.088457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:32.341073 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.340960    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"]
Nov 03 20:02:32.354653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.354618    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"]
Nov 03 20:02:32.354808 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.354749    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:32.435044 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.434996    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz57z\" (UniqueName: \"kubernetes.io/projected/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-kube-api-access-qz57z\") pod \"maas-api-revoke-keys-e2e-inf-a-225668-grjsp\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:32.435361 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.435336    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-225668-grjsp\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:32.435456 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.435382    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-225668-grjsp\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:32.537556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.537509    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz57z\" (UniqueName: \"kubernetes.io/projected/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-kube-api-access-qz57z\") pod \"maas-api-revoke-keys-e2e-inf-a-225668-grjsp\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:32.537844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.537821    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-225668-grjsp\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:32.537941 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.537902    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-225668-grjsp\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:32.538662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.538639    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-225668-grjsp\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:32.543946 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.543915    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-225668-grjsp\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:32.545695 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.545643    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz57z\" (UniqueName: \"kubernetes.io/projected/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-kube-api-access-qz57z\") pod \"maas-api-revoke-keys-e2e-inf-a-225668-grjsp\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:32.697758 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.697719    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:32.908958 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:32.908917    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:33.052481 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.051666    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:33.052879 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:02:33.052830    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553dba5d_e786_4e1f_b7c4_7a30cde0d87d.slice/crio-c33872bae52bf798bda90f9e0026729297763f32696f4adc19c384877026cc2d WatchSource:0}: Error finding container c33872bae52bf798bda90f9e0026729297763f32696f4adc19c384877026cc2d: Status 404 returned error can't find the container with id c33872bae52bf798bda90f9e0026729297763f32696f4adc19c384877026cc2d
Nov 03 20:02:33.057459 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.056520    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"]
Nov 03 20:02:33.168797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.168753    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-openshift-ca\") pod \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") "
Nov 03 20:02:33.169028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.168873    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-xks-ca\") pod \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") "
Nov 03 20:02:33.169120 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.169099    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cad6d22-639f-45c0-a03f-b7e04095b3ad-kube-api-access-gzbqz\" (UniqueName: \"kubernetes.io/projected/4cad6d22-639f-45c0-a03f-b7e04095b3ad-kube-api-access-gzbqz\") pod \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\" (UID: \"4cad6d22-639f-45c0-a03f-b7e04095b3ad\") "
Nov 03 20:02:33.169426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.169397    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-openshift-ca" pod "4cad6d22-639f-45c0-a03f-b7e04095b3ad" (UID: "4cad6d22-639f-45c0-a03f-b7e04095b3ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:33.170389 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.170340    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:33.238980 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.238898    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-xks-ca" pod "4cad6d22-639f-45c0-a03f-b7e04095b3ad" (UID: "4cad6d22-639f-45c0-a03f-b7e04095b3ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:33.238980 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.238955    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cad6d22-639f-45c0-a03f-b7e04095b3ad-kube-api-access-gzbqz" pod "4cad6d22-639f-45c0-a03f-b7e04095b3ad" (UID: "4cad6d22-639f-45c0-a03f-b7e04095b3ad"). InnerVolumeSpecName "kube-api-access-gzbqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:33.272471 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.272435    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cad6d22-639f-45c0-a03f-b7e04095b3ad-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:33.272471 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.272470    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbqz\" (UniqueName: \"kubernetes.io/projected/4cad6d22-639f-45c0-a03f-b7e04095b3ad-kube-api-access-gzbqz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:33.714770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.714735    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"
Nov 03 20:02:33.715160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.714826    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj" event={"ID":"4cad6d22-639f-45c0-a03f-b7e04095b3ad","Type":"ContainerDied","Data":"05fead65f96ba947fcac7458c8ec15e655ab2e79f814d5dbcacab288309ab189"}
Nov 03 20:02:33.715160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.714888    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05fead65f96ba947fcac7458c8ec15e655ab2e79f814d5dbcacab288309ab189"
Nov 03 20:02:33.717137 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.717104    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp" event={"ID":"553dba5d-e786-4e1f-b7c4-7a30cde0d87d","Type":"ContainerStarted","Data":"aea1288817bdf2db7bdd12c4e26782d2aee244139e82e0518c6da7cdb55eaa03"}
Nov 03 20:02:33.717303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.717183    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp" event={"ID":"553dba5d-e786-4e1f-b7c4-7a30cde0d87d","Type":"ContainerStarted","Data":"c33872bae52bf798bda90f9e0026729297763f32696f4adc19c384877026cc2d"}
Nov 03 20:02:33.742209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.742132    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp" podStartSLOduration=1.742116404 podStartE2EDuration="1.742116404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:33.733602114 +0000 UTC m=+1867.003725410" watchObservedRunningTime="2026-11-03 20:02:33.742116404 +0000 UTC m=+1867.012239699"
Nov 03 20:02:33.823824 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:33.823717    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:34.113102 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.113056    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:02:34.126105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.126071    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 20:02:34.194967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.194924    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/505a4164-fe42-4574-aeb6-fbe39f93e798-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505a4164-fe42-4574-aeb6-fbe39f93e798-istiod-ca-cert\") pod \"505a4164-fe42-4574-aeb6-fbe39f93e798\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") "
Nov 03 20:02:34.195229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195007    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-podinfo\") pod \"505a4164-fe42-4574-aeb6-fbe39f93e798\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") "
Nov 03 20:02:34.195229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195074    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-socket\") pod \"505a4164-fe42-4574-aeb6-fbe39f93e798\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") "
Nov 03 20:02:34.195386 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195342    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505a4164-fe42-4574-aeb6-fbe39f93e798-istiod-ca-cert" pod "505a4164-fe42-4574-aeb6-fbe39f93e798" (UID: "505a4164-fe42-4574-aeb6-fbe39f93e798"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:34.195458 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195388    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-socket" pod "505a4164-fe42-4574-aeb6-fbe39f93e798" (UID: "505a4164-fe42-4574-aeb6-fbe39f93e798"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:34.195527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195469    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-kube-api-access-fc2dc\" (UniqueName: \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-kube-api-access-fc2dc\") pod \"505a4164-fe42-4574-aeb6-fbe39f93e798\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") "
Nov 03 20:02:34.195606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195531    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-token\" (UniqueName: \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-token\") pod \"b66385a3-91b7-4e59-afbd-d6a57110a267\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") "
Nov 03 20:02:34.195606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195558    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-certs\") pod \"505a4164-fe42-4574-aeb6-fbe39f93e798\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") "
Nov 03 20:02:34.195714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195619    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-data\") pod \"505a4164-fe42-4574-aeb6-fbe39f93e798\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") "
Nov 03 20:02:34.195714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195652    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-credential-socket\") pod \"505a4164-fe42-4574-aeb6-fbe39f93e798\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") "
Nov 03 20:02:34.195714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195671    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-socket\") pod \"b66385a3-91b7-4e59-afbd-d6a57110a267\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") "
Nov 03 20:02:34.195882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195827    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-token\" (UniqueName: \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-token\") pod \"505a4164-fe42-4574-aeb6-fbe39f93e798\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") "
Nov 03 20:02:34.195882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195865    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-certs\") pod \"b66385a3-91b7-4e59-afbd-d6a57110a267\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") "
Nov 03 20:02:34.195993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195900    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-envoy\") pod \"b66385a3-91b7-4e59-afbd-d6a57110a267\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") "
Nov 03 20:02:34.195993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.195928    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-data\") pod \"b66385a3-91b7-4e59-afbd-d6a57110a267\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") "
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.196134    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-credential-socket\") pod \"b66385a3-91b7-4e59-afbd-d6a57110a267\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") "
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.196431    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-certs" pod "b66385a3-91b7-4e59-afbd-d6a57110a267" (UID: "b66385a3-91b7-4e59-afbd-d6a57110a267"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.196562    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-envoy\") pod \"505a4164-fe42-4574-aeb6-fbe39f93e798\" (UID: \"505a4164-fe42-4574-aeb6-fbe39f93e798\") "
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.196586    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-socket" pod "b66385a3-91b7-4e59-afbd-d6a57110a267" (UID: "b66385a3-91b7-4e59-afbd-d6a57110a267"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.196597    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66385a3-91b7-4e59-afbd-d6a57110a267-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66385a3-91b7-4e59-afbd-d6a57110a267-istiod-ca-cert\") pod \"b66385a3-91b7-4e59-afbd-d6a57110a267\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") "
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.196629    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-kube-api-access-jp54h\" (UniqueName: \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-kube-api-access-jp54h\") pod \"b66385a3-91b7-4e59-afbd-d6a57110a267\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") "
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.196666    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-podinfo\") pod \"b66385a3-91b7-4e59-afbd-d6a57110a267\" (UID: \"b66385a3-91b7-4e59-afbd-d6a57110a267\") "
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.196806    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-certs" pod "505a4164-fe42-4574-aeb6-fbe39f93e798" (UID: "505a4164-fe42-4574-aeb6-fbe39f93e798"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.196819    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-credential-socket" pod "505a4164-fe42-4574-aeb6-fbe39f93e798" (UID: "505a4164-fe42-4574-aeb6-fbe39f93e798"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.197015    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-data" pod "505a4164-fe42-4574-aeb6-fbe39f93e798" (UID: "505a4164-fe42-4574-aeb6-fbe39f93e798"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.197024    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-credential-socket" pod "b66385a3-91b7-4e59-afbd-d6a57110a267" (UID: "b66385a3-91b7-4e59-afbd-d6a57110a267"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.197606    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-data" pod "b66385a3-91b7-4e59-afbd-d6a57110a267" (UID: "b66385a3-91b7-4e59-afbd-d6a57110a267"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.197897    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66385a3-91b7-4e59-afbd-d6a57110a267-istiod-ca-cert" pod "b66385a3-91b7-4e59-afbd-d6a57110a267" (UID: "b66385a3-91b7-4e59-afbd-d6a57110a267"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:34.198155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.197950    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-podinfo" pod "505a4164-fe42-4574-aeb6-fbe39f93e798" (UID: "505a4164-fe42-4574-aeb6-fbe39f93e798"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198222    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-kube-api-access-fc2dc" pod "505a4164-fe42-4574-aeb6-fbe39f93e798" (UID: "505a4164-fe42-4574-aeb6-fbe39f93e798"). InnerVolumeSpecName "kube-api-access-fc2dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198358    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198383    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198401    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198415    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66385a3-91b7-4e59-afbd-d6a57110a267-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198431    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/505a4164-fe42-4574-aeb6-fbe39f93e798-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198445    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198459    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198473    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198487    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198500    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.198835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.198515    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.200662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.200631    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-kube-api-access-jp54h" pod "b66385a3-91b7-4e59-afbd-d6a57110a267" (UID: "b66385a3-91b7-4e59-afbd-d6a57110a267"). InnerVolumeSpecName "kube-api-access-jp54h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:34.201262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.201235    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-token" pod "505a4164-fe42-4574-aeb6-fbe39f93e798" (UID: "505a4164-fe42-4574-aeb6-fbe39f93e798"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:34.201262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.201263    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-token" pod "b66385a3-91b7-4e59-afbd-d6a57110a267" (UID: "b66385a3-91b7-4e59-afbd-d6a57110a267"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:34.201486 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.201279    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-podinfo" pod "b66385a3-91b7-4e59-afbd-d6a57110a267" (UID: "b66385a3-91b7-4e59-afbd-d6a57110a267"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:34.201560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.201482    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-envoy" pod "505a4164-fe42-4574-aeb6-fbe39f93e798" (UID: "505a4164-fe42-4574-aeb6-fbe39f93e798"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:34.202586 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.202564    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-envoy" pod "b66385a3-91b7-4e59-afbd-d6a57110a267" (UID: "b66385a3-91b7-4e59-afbd-d6a57110a267"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:34.299925 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.299885    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.299925 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.299922    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp54h\" (UniqueName: \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-kube-api-access-jp54h\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.299925 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.299933    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.300210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.299945    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc2dc\" (UniqueName: \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-kube-api-access-fc2dc\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.300210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.299956    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.300210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.299966    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/505a4164-fe42-4574-aeb6-fbe39f93e798-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.300210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.299975    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66385a3-91b7-4e59-afbd-d6a57110a267-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:34.548203 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.546481    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:34.729936 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.729898    4531 generic.go:356] "Generic (PLEG): container finished" podID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerID="b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7" exitCode=0
Nov 03 20:02:34.730450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.729960    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"
Nov 03 20:02:34.730450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.729981    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" event={"ID":"b66385a3-91b7-4e59-afbd-d6a57110a267","Type":"ContainerDied","Data":"b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7"}
Nov 03 20:02:34.730450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.730054    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4" event={"ID":"b66385a3-91b7-4e59-afbd-d6a57110a267","Type":"ContainerDied","Data":"434ec3bdb17560c286e376a3c53c925658a17d741b57c116c2d4f0f159e07162"}
Nov 03 20:02:34.730450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.730091    4531 scope.go:122] "RemoveContainer" containerID="b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7"
Nov 03 20:02:34.733397 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.733366    4531 generic.go:356] "Generic (PLEG): container finished" podID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerID="be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052" exitCode=0
Nov 03 20:02:34.733576 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.733557    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" event={"ID":"505a4164-fe42-4574-aeb6-fbe39f93e798","Type":"ContainerDied","Data":"be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052"}
Nov 03 20:02:34.733628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.733614    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr" event={"ID":"505a4164-fe42-4574-aeb6-fbe39f93e798","Type":"ContainerDied","Data":"3bb4e7040293956c1f8fbc3a7280919d3f1d98cb1d527e65bf3fb5f9c81be6e2"}
Nov 03 20:02:34.733704 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.733688    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"
Nov 03 20:02:34.735806 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.735779    4531 generic.go:356] "Generic (PLEG): container finished" podID="553dba5d-e786-4e1f-b7c4-7a30cde0d87d" containerID="aea1288817bdf2db7bdd12c4e26782d2aee244139e82e0518c6da7cdb55eaa03" exitCode=0
Nov 03 20:02:34.735909 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.735854    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp" event={"ID":"553dba5d-e786-4e1f-b7c4-7a30cde0d87d","Type":"ContainerDied","Data":"aea1288817bdf2db7bdd12c4e26782d2aee244139e82e0518c6da7cdb55eaa03"}
Nov 03 20:02:34.744520 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.744494    4531 scope.go:122] "RemoveContainer" containerID="b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7"
Nov 03 20:02:34.744896 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:34.744872    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7\": container with ID starting with b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7 not found: ID does not exist" containerID="b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7"
Nov 03 20:02:34.744960 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.744904    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7"} err="failed to get container status \"b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7\": rpc error: code = NotFound desc = could not find container \"b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7\": container with ID starting with b5c968f0735b3024029fac1c5d386e252eaacb6592137d89bbb24eb1b1094ab7 not found: ID does not exist"
Nov 03 20:02:34.744960 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.744925    4531 scope.go:122] "RemoveContainer" containerID="be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052"
Nov 03 20:02:34.758376 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.758260    4531 scope.go:122] "RemoveContainer" containerID="3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee"
Nov 03 20:02:34.770814 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.770779    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"]
Nov 03 20:02:34.775645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.774579    4531 scope.go:122] "RemoveContainer" containerID="be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052"
Nov 03 20:02:34.775645 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:34.774966    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052\": container with ID starting with be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052 not found: ID does not exist" containerID="be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052"
Nov 03 20:02:34.775645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.775003    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052"} err="failed to get container status \"be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052\": rpc error: code = NotFound desc = could not find container \"be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052\": container with ID starting with be34793dc7805a15b57a61f7ca49ab71801e076188d40edf648f97abece3d052 not found: ID does not exist"
Nov 03 20:02:34.775645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.775028    4531 scope.go:122] "RemoveContainer" containerID="3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee"
Nov 03 20:02:34.775645 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:34.775491    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee\": container with ID starting with 3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee not found: ID does not exist" containerID="3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee"
Nov 03 20:02:34.775645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.775524    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee"} err="failed to get container status \"3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee\": rpc error: code = NotFound desc = could not find container \"3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee\": container with ID starting with 3a1ffb1df42e660699c21bf86a8f3a850d7e12ff305c7b3c819a5a72b0a4f2ee not found: ID does not exist"
Nov 03 20:02:34.782673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.782633    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-4mcc4"]
Nov 03 20:02:34.799899 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.799811    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"]
Nov 03 20:02:34.805933 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.805891    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ffe132-openshift-default-c4d7b6ddf-rvblr"]
Nov 03 20:02:34.807872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.807845    4531 scope.go:122] "RemoveContainer" containerID="a763bcde7924dadde28ab95c476bcf85944d27ca28292c6b59047268b2ea66f7"
Nov 03 20:02:34.864346 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.864308    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" path="/var/lib/kubelet/pods/505a4164-fe42-4574-aeb6-fbe39f93e798/volumes"
Nov 03 20:02:34.865043 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:34.865025    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" path="/var/lib/kubelet/pods/b66385a3-91b7-4e59-afbd-d6a57110a267/volumes"
Nov 03 20:02:35.268200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.267268    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:35.645902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.645844    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld" podUID="4a54f455-59f9-410c-8b6a-8c56f0828800" containerName="main" containerID="cri-o://323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087" gracePeriod=45
Nov 03 20:02:35.721494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.721454    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld" podUID="4a54f455-59f9-410c-8b6a-8c56f0828800" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": dial tcp 10.133.0.209:8000: connect: connection refused"
Nov 03 20:02:35.731691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.731652    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"]
Nov 03 20:02:35.732714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732693    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4cad6d22-639f-45c0-a03f-b7e04095b3ad" containerName="revoke-keys"
Nov 03 20:02:35.732768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732716    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cad6d22-639f-45c0-a03f-b7e04095b3ad" containerName="revoke-keys"
Nov 03 20:02:35.732768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732735    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy"
Nov 03 20:02:35.732768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732741    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy"
Nov 03 20:02:35.732768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732759    4531 cpu_manager.go:414] "containerMap: removing container" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy"
Nov 03 20:02:35.732768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732764    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy"
Nov 03 20:02:35.732917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732781    4531 cpu_manager.go:414] "containerMap: removing container" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy"
Nov 03 20:02:35.732917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732786    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy"
Nov 03 20:02:35.732982 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732955    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cad6d22-639f-45c0-a03f-b7e04095b3ad" containerName="revoke-keys"
Nov 03 20:02:35.732982 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732964    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66385a3-91b7-4e59-afbd-d6a57110a267" containerName="istio-proxy"
Nov 03 20:02:35.732982 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732974    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy"
Nov 03 20:02:35.733079 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.732982    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="505a4164-fe42-4574-aeb6-fbe39f93e798" containerName="istio-proxy"
Nov 03 20:02:35.737869 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.737839    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:35.745080 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.745039    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"]
Nov 03 20:02:35.827776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.827039    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cc16508-hdxqj\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:35.827776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.827543    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6pnn\" (UniqueName: \"kubernetes.io/projected/dc17f74e-f63e-4614-9926-be37b19614a3-kube-api-access-w6pnn\") pod \"maas-api-revoke-sub-4cc16508-hdxqj\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:35.827776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.827592    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cc16508-hdxqj\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:35.934463 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.930259    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6pnn\" (UniqueName: \"kubernetes.io/projected/dc17f74e-f63e-4614-9926-be37b19614a3-kube-api-access-w6pnn\") pod \"maas-api-revoke-sub-4cc16508-hdxqj\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:35.934463 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.930323    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cc16508-hdxqj\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:35.934463 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.930427    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cc16508-hdxqj\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:35.934463 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.931544    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cc16508-hdxqj\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:35.935479 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.935428    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cc16508-hdxqj\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:35.957784 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:35.957748    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6pnn\" (UniqueName: \"kubernetes.io/projected/dc17f74e-f63e-4614-9926-be37b19614a3-kube-api-access-w6pnn\") pod \"maas-api-revoke-sub-4cc16508-hdxqj\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:36.010350 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.010320    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:36.057331 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.057300    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 20:02:36.065167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.065133    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:36.149505 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.149464    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:36.153276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.150097    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-tmp-dir\") pod \"4a54f455-59f9-410c-8b6a-8c56f0828800\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") "
Nov 03 20:02:36.153276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.150137    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-home\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-home\") pod \"4a54f455-59f9-410c-8b6a-8c56f0828800\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") "
Nov 03 20:02:36.153276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.150205    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-xks-ca\") pod \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") "
Nov 03 20:02:36.153276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.150350    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a54f455-59f9-410c-8b6a-8c56f0828800-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a54f455-59f9-410c-8b6a-8c56f0828800-tls-certs\") pod \"4a54f455-59f9-410c-8b6a-8c56f0828800\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") "
Nov 03 20:02:36.153276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.150442    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-openshift-ca\") pod \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") "
Nov 03 20:02:36.153276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.150488    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a54f455-59f9-410c-8b6a-8c56f0828800-kube-api-access-9pd4r\" (UniqueName: \"kubernetes.io/projected/4a54f455-59f9-410c-8b6a-8c56f0828800-kube-api-access-9pd4r\") pod \"4a54f455-59f9-410c-8b6a-8c56f0828800\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") "
Nov 03 20:02:36.153276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.150563    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-dshm\") pod \"4a54f455-59f9-410c-8b6a-8c56f0828800\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") "
Nov 03 20:02:36.153276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.150654    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-kube-api-access-qz57z\" (UniqueName: \"kubernetes.io/projected/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-kube-api-access-qz57z\") pod \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\" (UID: \"553dba5d-e786-4e1f-b7c4-7a30cde0d87d\") "
Nov 03 20:02:36.153276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.150724    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-model-cache\") pod \"4a54f455-59f9-410c-8b6a-8c56f0828800\" (UID: \"4a54f455-59f9-410c-8b6a-8c56f0828800\") "
Nov 03 20:02:36.153276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.151738    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-home" pod "4a54f455-59f9-410c-8b6a-8c56f0828800" (UID: "4a54f455-59f9-410c-8b6a-8c56f0828800"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:36.153276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.152095    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-openshift-ca" pod "553dba5d-e786-4e1f-b7c4-7a30cde0d87d" (UID: "553dba5d-e786-4e1f-b7c4-7a30cde0d87d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:36.209809 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.157351    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-tmp-dir" pod "4a54f455-59f9-410c-8b6a-8c56f0828800" (UID: "4a54f455-59f9-410c-8b6a-8c56f0828800"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:36.209809 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.157627    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-model-cache" pod "4a54f455-59f9-410c-8b6a-8c56f0828800" (UID: "4a54f455-59f9-410c-8b6a-8c56f0828800"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:36.227764 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.224334    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-dshm" pod "4a54f455-59f9-410c-8b6a-8c56f0828800" (UID: "4a54f455-59f9-410c-8b6a-8c56f0828800"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:36.230924 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.230889    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-xks-ca" pod "553dba5d-e786-4e1f-b7c4-7a30cde0d87d" (UID: "553dba5d-e786-4e1f-b7c4-7a30cde0d87d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:36.237233 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.233402    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-kube-api-access-qz57z" pod "553dba5d-e786-4e1f-b7c4-7a30cde0d87d" (UID: "553dba5d-e786-4e1f-b7c4-7a30cde0d87d"). InnerVolumeSpecName "kube-api-access-qz57z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:36.237233 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.233437    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a54f455-59f9-410c-8b6a-8c56f0828800-kube-api-access-9pd4r" pod "4a54f455-59f9-410c-8b6a-8c56f0828800" (UID: "4a54f455-59f9-410c-8b6a-8c56f0828800"). InnerVolumeSpecName "kube-api-access-9pd4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:36.237233 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.234428    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a54f455-59f9-410c-8b6a-8c56f0828800-tls-certs" pod "4a54f455-59f9-410c-8b6a-8c56f0828800" (UID: "4a54f455-59f9-410c-8b6a-8c56f0828800"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:36.253635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.253601    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.253635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.253632    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.253877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.253648    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.253877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.253657    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.253877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.253666    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a54f455-59f9-410c-8b6a-8c56f0828800-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.253877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.253676    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.253877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.253686    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pd4r\" (UniqueName: \"kubernetes.io/projected/4a54f455-59f9-410c-8b6a-8c56f0828800-kube-api-access-9pd4r\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.253877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.253696    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a54f455-59f9-410c-8b6a-8c56f0828800-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.253877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.253704    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz57z\" (UniqueName: \"kubernetes.io/projected/553dba5d-e786-4e1f-b7c4-7a30cde0d87d-kube-api-access-qz57z\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.498299 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:02:36.498257    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc17f74e_f63e_4614_9926_be37b19614a3.slice/crio-d8d42736e974b591a4fcf486621d1d1ea009abbd90567781f3f9dbe64b03191e WatchSource:0}: Error finding container d8d42736e974b591a4fcf486621d1d1ea009abbd90567781f3f9dbe64b03191e: Status 404 returned error can't find the container with id d8d42736e974b591a4fcf486621d1d1ea009abbd90567781f3f9dbe64b03191e
Nov 03 20:02:36.498759 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.498731    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"]
Nov 03 20:02:36.581307 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.581262    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main" containerID="cri-o://992d93fc660d58139c7add18026e6b68dcc7111876570f7f63b41022517fdcad" gracePeriod=45
Nov 03 20:02:36.770394 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.770346    4531 generic.go:356] "Generic (PLEG): container finished" podID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerID="992d93fc660d58139c7add18026e6b68dcc7111876570f7f63b41022517fdcad" exitCode=0
Nov 03 20:02:36.770918 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.770434    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" event={"ID":"a4699bd7-90b4-477c-bc1f-c819a5675de2","Type":"ContainerDied","Data":"992d93fc660d58139c7add18026e6b68dcc7111876570f7f63b41022517fdcad"}
Nov 03 20:02:36.772579 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.772545    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj" event={"ID":"dc17f74e-f63e-4614-9926-be37b19614a3","Type":"ContainerStarted","Data":"d8d42736e974b591a4fcf486621d1d1ea009abbd90567781f3f9dbe64b03191e"}
Nov 03 20:02:36.774570 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.774538    4531 generic.go:356] "Generic (PLEG): container finished" podID="4a54f455-59f9-410c-8b6a-8c56f0828800" containerID="323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087" exitCode=0
Nov 03 20:02:36.774718 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.774697    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld" event={"ID":"4a54f455-59f9-410c-8b6a-8c56f0828800","Type":"ContainerDied","Data":"323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087"}
Nov 03 20:02:36.774780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.774740    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld" event={"ID":"4a54f455-59f9-410c-8b6a-8c56f0828800","Type":"ContainerDied","Data":"d9a21e4a9b15ff5ccb065d55d088369f51bbec3013c2af30dd45d2aa3c1dc05c"}
Nov 03 20:02:36.774780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.774762    4531 scope.go:122] "RemoveContainer" containerID="323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087"
Nov 03 20:02:36.774889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.774874    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"
Nov 03 20:02:36.794635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.793240    4531 scope.go:122] "RemoveContainer" containerID="323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087"
Nov 03 20:02:36.795071 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:36.795031    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087\": container with ID starting with 323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087 not found: ID does not exist" containerID="323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087"
Nov 03 20:02:36.795494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.795078    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087"} err="failed to get container status \"323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087\": rpc error: code = NotFound desc = could not find container \"323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087\": container with ID starting with 323a97905404bea2db3bb45a052039a2f0b78d1643f19ed884b5baf25faa0087 not found: ID does not exist"
Nov 03 20:02:36.795494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.795154    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"
Nov 03 20:02:36.795494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.795156    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp" event={"ID":"553dba5d-e786-4e1f-b7c4-7a30cde0d87d","Type":"ContainerDied","Data":"c33872bae52bf798bda90f9e0026729297763f32696f4adc19c384877026cc2d"}
Nov 03 20:02:36.795494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.795252    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c33872bae52bf798bda90f9e0026729297763f32696f4adc19c384877026cc2d"
Nov 03 20:02:36.807030 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.806995    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"]
Nov 03 20:02:36.817625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.817576    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-225668/test-model-225668-kserve-6fcc8c56df-ltgld"]
Nov 03 20:02:36.883517 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.883479    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a54f455-59f9-410c-8b6a-8c56f0828800" path="/var/lib/kubelet/pods/4a54f455-59f9-410c-8b6a-8c56f0828800/volumes"
Nov 03 20:02:36.928708 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:36.928674    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 20:02:37.069643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.069546    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4699bd7-90b4-477c-bc1f-c819a5675de2-kube-api-access-4c5vw\" (UniqueName: \"kubernetes.io/projected/a4699bd7-90b4-477c-bc1f-c819a5675de2-kube-api-access-4c5vw\") pod \"a4699bd7-90b4-477c-bc1f-c819a5675de2\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") "
Nov 03 20:02:37.069643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.069631    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-tmp-dir\") pod \"a4699bd7-90b4-477c-bc1f-c819a5675de2\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") "
Nov 03 20:02:37.070131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.069738    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4699bd7-90b4-477c-bc1f-c819a5675de2-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4699bd7-90b4-477c-bc1f-c819a5675de2-tls-certs\") pod \"a4699bd7-90b4-477c-bc1f-c819a5675de2\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") "
Nov 03 20:02:37.070131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.069777    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-dshm\") pod \"a4699bd7-90b4-477c-bc1f-c819a5675de2\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") "
Nov 03 20:02:37.070131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.069800    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-home\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-home\") pod \"a4699bd7-90b4-477c-bc1f-c819a5675de2\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") "
Nov 03 20:02:37.070131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.069871    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-model-cache\") pod \"a4699bd7-90b4-477c-bc1f-c819a5675de2\" (UID: \"a4699bd7-90b4-477c-bc1f-c819a5675de2\") "
Nov 03 20:02:37.070131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.069892    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-tmp-dir" pod "a4699bd7-90b4-477c-bc1f-c819a5675de2" (UID: "a4699bd7-90b4-477c-bc1f-c819a5675de2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:37.070562 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.070539    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-home" pod "a4699bd7-90b4-477c-bc1f-c819a5675de2" (UID: "a4699bd7-90b4-477c-bc1f-c819a5675de2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:37.070810 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.070787    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:37.070877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.070816    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:37.070925 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.070873    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-model-cache" pod "a4699bd7-90b4-477c-bc1f-c819a5675de2" (UID: "a4699bd7-90b4-477c-bc1f-c819a5675de2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:37.072522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.072496    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4699bd7-90b4-477c-bc1f-c819a5675de2-tls-certs" pod "a4699bd7-90b4-477c-bc1f-c819a5675de2" (UID: "a4699bd7-90b4-477c-bc1f-c819a5675de2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:37.072665 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.072524    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-dshm" pod "a4699bd7-90b4-477c-bc1f-c819a5675de2" (UID: "a4699bd7-90b4-477c-bc1f-c819a5675de2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:37.072665 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.072638    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4699bd7-90b4-477c-bc1f-c819a5675de2-kube-api-access-4c5vw" pod "a4699bd7-90b4-477c-bc1f-c819a5675de2" (UID: "a4699bd7-90b4-477c-bc1f-c819a5675de2"). InnerVolumeSpecName "kube-api-access-4c5vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:37.134520 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.134476    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:37.187193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.186742    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5vw\" (UniqueName: \"kubernetes.io/projected/a4699bd7-90b4-477c-bc1f-c819a5675de2-kube-api-access-4c5vw\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:37.187193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.186793    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4699bd7-90b4-477c-bc1f-c819a5675de2-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:37.187193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.186810    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:37.187193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.186822    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4699bd7-90b4-477c-bc1f-c819a5675de2-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:37.720040 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:37.720002    4531 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc17f74e_f63e_4614_9926_be37b19614a3.slice/crio-ca32d590779be2edab6a6e74c4e9175119dd5cd9149eaffa86a449aed8c2bec1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc17f74e_f63e_4614_9926_be37b19614a3.slice/crio-conmon-ca32d590779be2edab6a6e74c4e9175119dd5cd9149eaffa86a449aed8c2bec1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:02:37.808086 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.808053    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"
Nov 03 20:02:37.808579 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.808052    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs" event={"ID":"a4699bd7-90b4-477c-bc1f-c819a5675de2","Type":"ContainerDied","Data":"5469a9d834cd8e15ad968c970a27b4a29e994eadc32b8e31f9cf1aa21f277d17"}
Nov 03 20:02:37.808579 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.808232    4531 scope.go:122] "RemoveContainer" containerID="992d93fc660d58139c7add18026e6b68dcc7111876570f7f63b41022517fdcad"
Nov 03 20:02:37.810349 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.810318    4531 generic.go:356] "Generic (PLEG): container finished" podID="dc17f74e-f63e-4614-9926-be37b19614a3" containerID="ca32d590779be2edab6a6e74c4e9175119dd5cd9149eaffa86a449aed8c2bec1" exitCode=0
Nov 03 20:02:37.810480 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.810354    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj" event={"ID":"dc17f74e-f63e-4614-9926-be37b19614a3","Type":"ContainerDied","Data":"ca32d590779be2edab6a6e74c4e9175119dd5cd9149eaffa86a449aed8c2bec1"}
Nov 03 20:02:37.890874 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.888106    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:37.918278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.914262    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"]
Nov 03 20:02:37.922799 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:37.922759    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-102379/simulator-w6-102379-kserve-8cd95d64c-m67zs"]
Nov 03 20:02:38.539902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:38.539863    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:38.865844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:38.865806    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" path="/var/lib/kubelet/pods/a4699bd7-90b4-477c-bc1f-c819a5675de2/volumes"
Nov 03 20:02:39.024942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.024911    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:39.121359 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.121264    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-xks-ca\") pod \"dc17f74e-f63e-4614-9926-be37b19614a3\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") "
Nov 03 20:02:39.121540 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.121506    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc17f74e-f63e-4614-9926-be37b19614a3-kube-api-access-w6pnn\" (UniqueName: \"kubernetes.io/projected/dc17f74e-f63e-4614-9926-be37b19614a3-kube-api-access-w6pnn\") pod \"dc17f74e-f63e-4614-9926-be37b19614a3\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") "
Nov 03 20:02:39.121673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.121655    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-openshift-ca\") pod \"dc17f74e-f63e-4614-9926-be37b19614a3\" (UID: \"dc17f74e-f63e-4614-9926-be37b19614a3\") "
Nov 03 20:02:39.122154 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.122124    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-openshift-ca" pod "dc17f74e-f63e-4614-9926-be37b19614a3" (UID: "dc17f74e-f63e-4614-9926-be37b19614a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:39.122590 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.122569    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:39.123731 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.123697    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-xks-ca" pod "dc17f74e-f63e-4614-9926-be37b19614a3" (UID: "dc17f74e-f63e-4614-9926-be37b19614a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:39.124242 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.124222    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc17f74e-f63e-4614-9926-be37b19614a3-kube-api-access-w6pnn" pod "dc17f74e-f63e-4614-9926-be37b19614a3" (UID: "dc17f74e-f63e-4614-9926-be37b19614a3"). InnerVolumeSpecName "kube-api-access-w6pnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:39.224604 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.224566    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6pnn\" (UniqueName: \"kubernetes.io/projected/dc17f74e-f63e-4614-9926-be37b19614a3-kube-api-access-w6pnn\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:39.224604 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.224601    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc17f74e-f63e-4614-9926-be37b19614a3-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:39.488383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.488327    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"]
Nov 03 20:02:39.519200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.516332    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"]
Nov 03 20:02:39.929199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.919604    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj" event={"ID":"dc17f74e-f63e-4614-9926-be37b19614a3","Type":"ContainerDied","Data":"d8d42736e974b591a4fcf486621d1d1ea009abbd90567781f3f9dbe64b03191e"}
Nov 03 20:02:39.929199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.919681    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8d42736e974b591a4fcf486621d1d1ea009abbd90567781f3f9dbe64b03191e"
Nov 03 20:02:39.929199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:39.919766    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"
Nov 03 20:02:40.470021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:40.469885    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:40.646198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:40.641139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:41.320112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:41.320065    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:42.005201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:42.004532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:42.597453 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:42.597404    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:42.729398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:42.729356    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:43.555549 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:43.555511    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:44.162044 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:44.162008    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:44.489146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:44.489037    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj" podUID="9f057237-c5bb-4f1e-b8e3-ae085f4f1b92" containerName="payload-processing" containerID="cri-o://0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa" gracePeriod=25
Nov 03 20:02:44.517672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:44.517625    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6" podUID="35af209c-d9d9-4821-a124-2183e18a84c5" containerName="payload-pre-processing" containerID="cri-o://8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0" gracePeriod=25
Nov 03 20:02:44.945892 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:44.945856    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:45.749196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:45.743826    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:46.519332 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:46.519267    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:47.039225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:47.039181    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:47.182166 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:47.182114    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:47.186389 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:47.186360    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"]
Nov 03 20:02:47.191634 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:47.191606    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-225668-grjsp"]
Nov 03 20:02:47.353253 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:47.349711    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:47.735199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:47.735157    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:48.610621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:48.610579    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:48.861895 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:48.861799    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="553dba5d-e786-4e1f-b7c4-7a30cde0d87d" path="/var/lib/kubelet/pods/553dba5d-e786-4e1f-b7c4-7a30cde0d87d/volumes"
Nov 03 20:02:49.314003 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:49.313964    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:49.485271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:49.485230    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:49.532701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:49.532661    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"]
Nov 03 20:02:49.537552 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:49.537523    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a99fad-kwngj"]
Nov 03 20:02:49.653225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:49.651539    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:49.942696 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:49.942601    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"]
Nov 03 20:02:49.954198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:49.943154    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy" containerID="cri-o://d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421" gracePeriod=30
Nov 03 20:02:50.009449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:50.008470    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"]
Nov 03 20:02:50.009449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:50.008802    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy" containerID="cri-o://059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d" gracePeriod=30
Nov 03 20:02:50.086185 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:50.086029    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:50.576571 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:50.576526    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:02:50.842261 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:50.835325    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:50.903544 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:50.903507    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cad6d22-639f-45c0-a03f-b7e04095b3ad" path="/var/lib/kubelet/pods/4cad6d22-639f-45c0-a03f-b7e04095b3ad/volumes"
Nov 03 20:02:51.305861 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:51.305827    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:51.770475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:51.770441    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:52.448872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:52.448242    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:52.865251 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:52.855122    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:52.890805 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:52.890741    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:02:52.894398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:52.894361    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 20:02:53.296799 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:53.275051    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:53.481212 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:53.479264    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:53.689711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:53.689668    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj" podUID="9f057237-c5bb-4f1e-b8e3-ae085f4f1b92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:53.689711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:53.689689    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6" podUID="35af209c-d9d9-4821-a124-2183e18a84c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:54.135672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:54.135638    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:54.271764 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:54.271727    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:54.407646 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:54.407557    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"]
Nov 03 20:02:54.407851 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:54.407823    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" podUID="74560e30-953f-48f5-8031-9a73b7218cef" containerName="istio-proxy" containerID="cri-o://75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37" gracePeriod=30
Nov 03 20:02:54.524069 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:54.524007    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:55.268696 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.268593    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"]
Nov 03 20:02:55.269872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.269841    4531 cpu_manager.go:414] "containerMap: removing container" podUID="553dba5d-e786-4e1f-b7c4-7a30cde0d87d" containerName="revoke-keys"
Nov 03 20:02:55.269872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.269872    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="553dba5d-e786-4e1f-b7c4-7a30cde0d87d" containerName="revoke-keys"
Nov 03 20:02:55.270100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.269892    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main"
Nov 03 20:02:55.270100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.269900    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main"
Nov 03 20:02:55.270100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.269958    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dc17f74e-f63e-4614-9926-be37b19614a3" containerName="revoke-keys"
Nov 03 20:02:55.270100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.269966    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc17f74e-f63e-4614-9926-be37b19614a3" containerName="revoke-keys"
Nov 03 20:02:55.270100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.269986    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4a54f455-59f9-410c-8b6a-8c56f0828800" containerName="main"
Nov 03 20:02:55.270100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.269994    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a54f455-59f9-410c-8b6a-8c56f0828800" containerName="main"
Nov 03 20:02:55.271443 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.270232    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="553dba5d-e786-4e1f-b7c4-7a30cde0d87d" containerName="revoke-keys"
Nov 03 20:02:55.271443 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.270247    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4699bd7-90b4-477c-bc1f-c819a5675de2" containerName="main"
Nov 03 20:02:55.271443 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.270266    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc17f74e-f63e-4614-9926-be37b19614a3" containerName="revoke-keys"
Nov 03 20:02:55.271443 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.270278    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a54f455-59f9-410c-8b6a-8c56f0828800" containerName="main"
Nov 03 20:02:55.278728 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.278704    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:55.295253 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.294995    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:02:55.295253 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.295075    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:02:55.297995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.297959    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"]
Nov 03 20:02:55.374194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.374154    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 20:02:55.421473 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.421437    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:55.421772 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.421751    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vgdv\" (UniqueName: \"kubernetes.io/projected/b0235d15-197e-47e9-b122-e6e49eefbe9f-kube-api-access-2vgdv\") pod \"maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:55.421920 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.421903    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:55.449586 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.449558    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:55.450775 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.450755    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 20:02:55.522901 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.522860    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c80951b-0a70-422b-acab-00b767b5304f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c80951b-0a70-422b-acab-00b767b5304f-istio-podinfo\") pod \"7c80951b-0a70-422b-acab-00b767b5304f\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") "
Nov 03 20:02:55.523265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523238    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9883aa-fca8-416e-8667-85354d4c2702-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9883aa-fca8-416e-8667-85354d4c2702-istiod-ca-cert\") pod \"dd9883aa-fca8-416e-8667-85354d4c2702\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") "
Nov 03 20:02:55.523430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523413    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c80951b-0a70-422b-acab-00b767b5304f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c80951b-0a70-422b-acab-00b767b5304f-istiod-ca-cert\") pod \"7c80951b-0a70-422b-acab-00b767b5304f\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") "
Nov 03 20:02:55.523530 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523464    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-istio-token\" (UniqueName: \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-istio-token\") pod \"7c80951b-0a70-422b-acab-00b767b5304f\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") "
Nov 03 20:02:55.523608 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523542    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-istio-token\" (UniqueName: \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-istio-token\") pod \"dd9883aa-fca8-416e-8667-85354d4c2702\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") "
Nov 03 20:02:55.523608 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523584    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-envoy\") pod \"dd9883aa-fca8-416e-8667-85354d4c2702\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") "
Nov 03 20:02:55.523726 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523662    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-kube-api-access-b4znp\" (UniqueName: \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-kube-api-access-b4znp\") pod \"7c80951b-0a70-422b-acab-00b767b5304f\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") "
Nov 03 20:02:55.523726 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523700    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-kube-api-access-hfl6h\" (UniqueName: \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-kube-api-access-hfl6h\") pod \"dd9883aa-fca8-416e-8667-85354d4c2702\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") "
Nov 03 20:02:55.523839 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523761    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-data\") pod \"dd9883aa-fca8-416e-8667-85354d4c2702\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") "
Nov 03 20:02:55.523839 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523812    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-data\") pod \"7c80951b-0a70-422b-acab-00b767b5304f\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") "
Nov 03 20:02:55.523947 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523896    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-socket\") pod \"7c80951b-0a70-422b-acab-00b767b5304f\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") "
Nov 03 20:02:55.524003 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523984    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-credential-socket\") pod \"7c80951b-0a70-422b-acab-00b767b5304f\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") "
Nov 03 20:02:55.524128 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.524041    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-socket\") pod \"dd9883aa-fca8-416e-8667-85354d4c2702\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") "
Nov 03 20:02:55.524128 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.524109    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd9883aa-fca8-416e-8667-85354d4c2702-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9883aa-fca8-416e-8667-85354d4c2702-istio-podinfo\") pod \"dd9883aa-fca8-416e-8667-85354d4c2702\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") "
Nov 03 20:02:55.524565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.523753    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9883aa-fca8-416e-8667-85354d4c2702-istiod-ca-cert" pod "dd9883aa-fca8-416e-8667-85354d4c2702" (UID: "dd9883aa-fca8-416e-8667-85354d4c2702"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:55.524565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.524143    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-certs\") pod \"dd9883aa-fca8-416e-8667-85354d4c2702\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") "
Nov 03 20:02:55.524565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.524323    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-credential-socket\") pod \"dd9883aa-fca8-416e-8667-85354d4c2702\" (UID: \"dd9883aa-fca8-416e-8667-85354d4c2702\") "
Nov 03 20:02:55.524565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.524435    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-certs\") pod \"7c80951b-0a70-422b-acab-00b767b5304f\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") "
Nov 03 20:02:55.524565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.524499    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-envoy\") pod \"7c80951b-0a70-422b-acab-00b767b5304f\" (UID: \"7c80951b-0a70-422b-acab-00b767b5304f\") "
Nov 03 20:02:55.524935 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.524910    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:55.525024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.525009    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:55.525428 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.525409    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vgdv\" (UniqueName: \"kubernetes.io/projected/b0235d15-197e-47e9-b122-e6e49eefbe9f-kube-api-access-2vgdv\") pod \"maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:55.525590 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.525576    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9883aa-fca8-416e-8667-85354d4c2702-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.526218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.524435    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-credential-socket" pod "7c80951b-0a70-422b-acab-00b767b5304f" (UID: "7c80951b-0a70-422b-acab-00b767b5304f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.526328 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.524556    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-data" pod "7c80951b-0a70-422b-acab-00b767b5304f" (UID: "7c80951b-0a70-422b-acab-00b767b5304f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.526328 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.524715    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-socket" pod "7c80951b-0a70-422b-acab-00b767b5304f" (UID: "7c80951b-0a70-422b-acab-00b767b5304f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.526328 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.524910    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-certs" pod "dd9883aa-fca8-416e-8667-85354d4c2702" (UID: "dd9883aa-fca8-416e-8667-85354d4c2702"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.526328 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.525114    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-socket" pod "dd9883aa-fca8-416e-8667-85354d4c2702" (UID: "dd9883aa-fca8-416e-8667-85354d4c2702"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.526328 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.525137    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c80951b-0a70-422b-acab-00b767b5304f-istiod-ca-cert" pod "7c80951b-0a70-422b-acab-00b767b5304f" (UID: "7c80951b-0a70-422b-acab-00b767b5304f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:55.526328 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.525987    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-credential-socket" pod "dd9883aa-fca8-416e-8667-85354d4c2702" (UID: "dd9883aa-fca8-416e-8667-85354d4c2702"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.526328 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.526155    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-certs" pod "7c80951b-0a70-422b-acab-00b767b5304f" (UID: "7c80951b-0a70-422b-acab-00b767b5304f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.526870 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.526845    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-istio-token" pod "dd9883aa-fca8-416e-8667-85354d4c2702" (UID: "dd9883aa-fca8-416e-8667-85354d4c2702"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.529227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.527145    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-data" pod "dd9883aa-fca8-416e-8667-85354d4c2702" (UID: "dd9883aa-fca8-416e-8667-85354d4c2702"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.529227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.528673    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:55.529464 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.529439    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-envoy" pod "7c80951b-0a70-422b-acab-00b767b5304f" (UID: "7c80951b-0a70-422b-acab-00b767b5304f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.530704 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.530290    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:55.530704 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.530580    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c80951b-0a70-422b-acab-00b767b5304f-istio-podinfo" pod "7c80951b-0a70-422b-acab-00b767b5304f" (UID: "7c80951b-0a70-422b-acab-00b767b5304f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:55.532950 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.532931    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vgdv\" (UniqueName: \"kubernetes.io/projected/b0235d15-197e-47e9-b122-e6e49eefbe9f-kube-api-access-2vgdv\") pod \"maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:55.536612 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.536548    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd9883aa-fca8-416e-8667-85354d4c2702-istio-podinfo" pod "dd9883aa-fca8-416e-8667-85354d4c2702" (UID: "dd9883aa-fca8-416e-8667-85354d4c2702"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:55.536943 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.536610    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-istio-token" pod "7c80951b-0a70-422b-acab-00b767b5304f" (UID: "7c80951b-0a70-422b-acab-00b767b5304f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.536943 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.536634    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-envoy" pod "dd9883aa-fca8-416e-8667-85354d4c2702" (UID: "dd9883aa-fca8-416e-8667-85354d4c2702"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.537402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.537377    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-kube-api-access-b4znp" pod "7c80951b-0a70-422b-acab-00b767b5304f" (UID: "7c80951b-0a70-422b-acab-00b767b5304f"). InnerVolumeSpecName "kube-api-access-b4znp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.538610 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.538588    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-kube-api-access-hfl6h" pod "dd9883aa-fca8-416e-8667-85354d4c2702" (UID: "dd9883aa-fca8-416e-8667-85354d4c2702"). InnerVolumeSpecName "kube-api-access-hfl6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.626577 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.625342    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:55.636142 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.636098    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.636475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.636439    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c80951b-0a70-422b-acab-00b767b5304f-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.636634 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.636618    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c80951b-0a70-422b-acab-00b767b5304f-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.636739 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.636725    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.636882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.636865    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.636976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.636963    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.637071 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.637059    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4znp\" (UniqueName: \"kubernetes.io/projected/7c80951b-0a70-422b-acab-00b767b5304f-kube-api-access-b4znp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.637151 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.637140    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfl6h\" (UniqueName: \"kubernetes.io/projected/dd9883aa-fca8-416e-8667-85354d4c2702-kube-api-access-hfl6h\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.637340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.637314    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.637519 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.637461    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.637671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.637650    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.637796 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.637780    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.642867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.642836    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.642867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.642867    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9883aa-fca8-416e-8667-85354d4c2702-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.643074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.642882    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.643074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.642896    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9883aa-fca8-416e-8667-85354d4c2702-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.643074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.642930    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c80951b-0a70-422b-acab-00b767b5304f-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.679865 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.679819    4531 generic.go:356] "Generic (PLEG): container finished" podID="7c80951b-0a70-422b-acab-00b767b5304f" containerID="059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d" exitCode=0
Nov 03 20:02:55.680281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.680248    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" event={"ID":"7c80951b-0a70-422b-acab-00b767b5304f","Type":"ContainerDied","Data":"059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d"}
Nov 03 20:02:55.680461 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.680381    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828" event={"ID":"7c80951b-0a70-422b-acab-00b767b5304f","Type":"ContainerDied","Data":"e8f742231d1dd76850cbc4d1e28826d028b159e9c0a9bd11ad29a3b821f478d1"}
Nov 03 20:02:55.680461 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.680431    4531 scope.go:122] "RemoveContainer" containerID="059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d"
Nov 03 20:02:55.680770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.680746    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"
Nov 03 20:02:55.682346 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.682307    4531 generic.go:356] "Generic (PLEG): container finished" podID="dd9883aa-fca8-416e-8667-85354d4c2702" containerID="d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421" exitCode=0
Nov 03 20:02:55.682510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.682361    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" event={"ID":"dd9883aa-fca8-416e-8667-85354d4c2702","Type":"ContainerDied","Data":"d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421"}
Nov 03 20:02:55.682510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.682396    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9" event={"ID":"dd9883aa-fca8-416e-8667-85354d4c2702","Type":"ContainerDied","Data":"f2ddc1f6ec24483e9923a96c4dfdf99e60c694ccc8169446a4f1de4e5b63c045"}
Nov 03 20:02:55.682510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.682441    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"
Nov 03 20:02:55.696053 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.696018    4531 scope.go:122] "RemoveContainer" containerID="059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d"
Nov 03 20:02:55.696508 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:55.696476    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d\": container with ID starting with 059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d not found: ID does not exist" containerID="059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d"
Nov 03 20:02:55.696667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.696520    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d"} err="failed to get container status \"059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d\": rpc error: code = NotFound desc = could not find container \"059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d\": container with ID starting with 059d1458845b310331b88caa3d2e46607c2b99c3185bbab49206bd74730f4b7d not found: ID does not exist"
Nov 03 20:02:55.696667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.696542    4531 scope.go:122] "RemoveContainer" containerID="d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421"
Nov 03 20:02:55.731242 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.726359    4531 scope.go:122] "RemoveContainer" containerID="d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421"
Nov 03 20:02:55.731242 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:55.730416    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421\": container with ID starting with d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421 not found: ID does not exist" containerID="d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421"
Nov 03 20:02:55.731242 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.730456    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421"} err="failed to get container status \"d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421\": rpc error: code = NotFound desc = could not find container \"d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421\": container with ID starting with d61df1fa387cfba1492fcad1ce5f2a86c0e807642300ff997b130fdcce67c421 not found: ID does not exist"
Nov 03 20:02:55.734533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.734502    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"]
Nov 03 20:02:55.739738 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.739709    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-p4828"]
Nov 03 20:02:55.746243 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.746167    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"]
Nov 03 20:02:55.753195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.753121    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-d6nh9"]
Nov 03 20:02:55.987284 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:55.987249    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"]
Nov 03 20:02:55.987819 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:02:55.987789    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0235d15_197e_47e9_b122_e6e49eefbe9f.slice/crio-7821623fa4b74b259a07b9cbfaf74492ebde9a7928d7a0a2e13dc38960e35939 WatchSource:0}: Error finding container 7821623fa4b74b259a07b9cbfaf74492ebde9a7928d7a0a2e13dc38960e35939: Status 404 returned error can't find the container with id 7821623fa4b74b259a07b9cbfaf74492ebde9a7928d7a0a2e13dc38960e35939
Nov 03 20:02:56.261390 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:56.257416    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.530514 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:56.530421    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.665727 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:56.665696    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.694757 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:56.694720    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb" event={"ID":"b0235d15-197e-47e9-b122-e6e49eefbe9f","Type":"ContainerStarted","Data":"d45f142b74deb59662e0aae05ee90558c54f0d711eaa4b1e64c0163be11084b2"}
Nov 03 20:02:56.694957 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:56.694787    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb" event={"ID":"b0235d15-197e-47e9-b122-e6e49eefbe9f","Type":"ContainerStarted","Data":"7821623fa4b74b259a07b9cbfaf74492ebde9a7928d7a0a2e13dc38960e35939"}
Nov 03 20:02:56.707995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:56.707939    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb" podStartSLOduration=1.707921595 podStartE2EDuration="1.707921595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:56.707295949 +0000 UTC m=+1889.977419245" watchObservedRunningTime="2026-11-03 20:02:56.707921595 +0000 UTC m=+1889.978044889"
Nov 03 20:02:56.862469 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:56.862391    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c80951b-0a70-422b-acab-00b767b5304f" path="/var/lib/kubelet/pods/7c80951b-0a70-422b-acab-00b767b5304f/volumes"
Nov 03 20:02:56.863010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:56.862995    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" path="/var/lib/kubelet/pods/dd9883aa-fca8-416e-8667-85354d4c2702/volumes"
Nov 03 20:02:56.910196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:56.897811    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:57.028481 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:57.028446    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:57.238786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:57.238741    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:57.405429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:57.405395    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:57.707319 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:57.707281    4531 generic.go:356] "Generic (PLEG): container finished" podID="b0235d15-197e-47e9-b122-e6e49eefbe9f" containerID="d45f142b74deb59662e0aae05ee90558c54f0d711eaa4b1e64c0163be11084b2" exitCode=0
Nov 03 20:02:57.707782 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:57.707374    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb" event={"ID":"b0235d15-197e-47e9-b122-e6e49eefbe9f","Type":"ContainerDied","Data":"d45f142b74deb59662e0aae05ee90558c54f0d711eaa4b1e64c0163be11084b2"}
Nov 03 20:02:57.883311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:57.882501    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:58.089262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:58.086047    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:58.491202 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:58.490992    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:58.659205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:58.653697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:59.085486 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.085453    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:59.158764 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.158737    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:59.281823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.281735    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0235d15-197e-47e9-b122-e6e49eefbe9f-kube-api-access-2vgdv\" (UniqueName: \"kubernetes.io/projected/b0235d15-197e-47e9-b122-e6e49eefbe9f-kube-api-access-2vgdv\") pod \"b0235d15-197e-47e9-b122-e6e49eefbe9f\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") "
Nov 03 20:02:59.282018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.281825    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-xks-ca\") pod \"b0235d15-197e-47e9-b122-e6e49eefbe9f\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") "
Nov 03 20:02:59.282018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.282004    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-openshift-ca\") pod \"b0235d15-197e-47e9-b122-e6e49eefbe9f\" (UID: \"b0235d15-197e-47e9-b122-e6e49eefbe9f\") "
Nov 03 20:02:59.282387 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.282360    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-openshift-ca" pod "b0235d15-197e-47e9-b122-e6e49eefbe9f" (UID: "b0235d15-197e-47e9-b122-e6e49eefbe9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:59.282681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.282660    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.283921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.283895    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0235d15-197e-47e9-b122-e6e49eefbe9f-kube-api-access-2vgdv" pod "b0235d15-197e-47e9-b122-e6e49eefbe9f" (UID: "b0235d15-197e-47e9-b122-e6e49eefbe9f"). InnerVolumeSpecName "kube-api-access-2vgdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:59.284063 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.283992    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-xks-ca" pod "b0235d15-197e-47e9-b122-e6e49eefbe9f" (UID: "b0235d15-197e-47e9-b122-e6e49eefbe9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:59.384883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.384839    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vgdv\" (UniqueName: \"kubernetes.io/projected/b0235d15-197e-47e9-b122-e6e49eefbe9f-kube-api-access-2vgdv\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.384883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.384879    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0235d15-197e-47e9-b122-e6e49eefbe9f-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.825422 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.825391    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 20:02:59.934229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.930837    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74560e30-953f-48f5-8031-9a73b7218cef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74560e30-953f-48f5-8031-9a73b7218cef-istiod-ca-cert\") pod \"74560e30-953f-48f5-8031-9a73b7218cef\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") "
Nov 03 20:02:59.934229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.930924    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-kube-api-access-wdfkg\" (UniqueName: \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-kube-api-access-wdfkg\") pod \"74560e30-953f-48f5-8031-9a73b7218cef\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") "
Nov 03 20:02:59.934229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.931028    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-socket\") pod \"74560e30-953f-48f5-8031-9a73b7218cef\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") "
Nov 03 20:02:59.934229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.931165    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74560e30-953f-48f5-8031-9a73b7218cef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74560e30-953f-48f5-8031-9a73b7218cef-istio-podinfo\") pod \"74560e30-953f-48f5-8031-9a73b7218cef\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") "
Nov 03 20:02:59.934229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.931215    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-credential-socket\") pod \"74560e30-953f-48f5-8031-9a73b7218cef\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") "
Nov 03 20:02:59.934229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.931267    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-istio-token\" (UniqueName: \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-istio-token\") pod \"74560e30-953f-48f5-8031-9a73b7218cef\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") "
Nov 03 20:02:59.934229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.931316    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-envoy\") pod \"74560e30-953f-48f5-8031-9a73b7218cef\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") "
Nov 03 20:02:59.934229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.931446    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-certs\") pod \"74560e30-953f-48f5-8031-9a73b7218cef\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") "
Nov 03 20:02:59.934229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.931466    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-data\") pod \"74560e30-953f-48f5-8031-9a73b7218cef\" (UID: \"74560e30-953f-48f5-8031-9a73b7218cef\") "
Nov 03 20:02:59.934229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.932277    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-data" pod "74560e30-953f-48f5-8031-9a73b7218cef" (UID: "74560e30-953f-48f5-8031-9a73b7218cef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:59.934889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.934860    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74560e30-953f-48f5-8031-9a73b7218cef-istio-podinfo" pod "74560e30-953f-48f5-8031-9a73b7218cef" (UID: "74560e30-953f-48f5-8031-9a73b7218cef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:59.935077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.935060    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-credential-socket" pod "74560e30-953f-48f5-8031-9a73b7218cef" (UID: "74560e30-953f-48f5-8031-9a73b7218cef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:59.937092 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.937063    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-istio-token" pod "74560e30-953f-48f5-8031-9a73b7218cef" (UID: "74560e30-953f-48f5-8031-9a73b7218cef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:59.937092 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.937078    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74560e30-953f-48f5-8031-9a73b7218cef-istiod-ca-cert" pod "74560e30-953f-48f5-8031-9a73b7218cef" (UID: "74560e30-953f-48f5-8031-9a73b7218cef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:59.937314 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.937147    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-certs" pod "74560e30-953f-48f5-8031-9a73b7218cef" (UID: "74560e30-953f-48f5-8031-9a73b7218cef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:59.938460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.938397    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-socket" pod "74560e30-953f-48f5-8031-9a73b7218cef" (UID: "74560e30-953f-48f5-8031-9a73b7218cef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:59.939313 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.939288    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-envoy" pod "74560e30-953f-48f5-8031-9a73b7218cef" (UID: "74560e30-953f-48f5-8031-9a73b7218cef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:59.939425 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.939388    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-kube-api-access-wdfkg" pod "74560e30-953f-48f5-8031-9a73b7218cef" (UID: "74560e30-953f-48f5-8031-9a73b7218cef"). InnerVolumeSpecName "kube-api-access-wdfkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:59.945311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.945293    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"
Nov 03 20:02:59.945412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.945323    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb" event={"ID":"b0235d15-197e-47e9-b122-e6e49eefbe9f","Type":"ContainerDied","Data":"7821623fa4b74b259a07b9cbfaf74492ebde9a7928d7a0a2e13dc38960e35939"}
Nov 03 20:02:59.945412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.945380    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7821623fa4b74b259a07b9cbfaf74492ebde9a7928d7a0a2e13dc38960e35939"
Nov 03 20:02:59.947126 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.947100    4531 generic.go:356] "Generic (PLEG): container finished" podID="74560e30-953f-48f5-8031-9a73b7218cef" containerID="75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37" exitCode=0
Nov 03 20:02:59.947303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.947125    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" event={"ID":"74560e30-953f-48f5-8031-9a73b7218cef","Type":"ContainerDied","Data":"75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37"}
Nov 03 20:02:59.947303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.947143    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"
Nov 03 20:02:59.947303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.947257    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh" event={"ID":"74560e30-953f-48f5-8031-9a73b7218cef","Type":"ContainerDied","Data":"f3b82e8eaf6a47966545804fd9db6e4170b39cabc8d4ba7caf444b7f6ef3a389"}
Nov 03 20:02:59.947303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.947262    4531 scope.go:122] "RemoveContainer" containerID="75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37"
Nov 03 20:02:59.971287 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.971260    4531 scope.go:122] "RemoveContainer" containerID="75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37"
Nov 03 20:02:59.971704 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:02:59.971673    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37\": container with ID starting with 75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37 not found: ID does not exist" containerID="75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37"
Nov 03 20:02:59.971783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.971718    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37"} err="failed to get container status \"75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37\": rpc error: code = NotFound desc = could not find container \"75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37\": container with ID starting with 75f11ab2142ffadd4176351cb5990273c531dde9091acf4a65ed52ca88bd3e37 not found: ID does not exist"
Nov 03 20:02:59.988423 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.988389    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"]
Nov 03 20:02:59.997459 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:02:59.997424    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-2mjrh"]
Nov 03 20:03:00.034219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.034137    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:00.034219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.034197    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:00.034219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.034215    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74560e30-953f-48f5-8031-9a73b7218cef-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:00.034470 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.034231    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdfkg\" (UniqueName: \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-kube-api-access-wdfkg\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:00.034470 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.034248    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:00.034470 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.034261    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74560e30-953f-48f5-8031-9a73b7218cef-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:00.034470 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.034277    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:00.034470 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.034289    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74560e30-953f-48f5-8031-9a73b7218cef-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:00.034470 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.034301    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74560e30-953f-48f5-8031-9a73b7218cef-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:00.292719 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.292678    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:00.429243 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.427102    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:00.862671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:00.862591    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74560e30-953f-48f5-8031-9a73b7218cef" path="/var/lib/kubelet/pods/74560e30-953f-48f5-8031-9a73b7218cef/volumes"
Nov 03 20:03:01.077201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:01.073870    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:02.013200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:02.001953    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:02.676859 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:02.676828    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:02.801411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:02.801379    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:03.052134 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:03.052013    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:03.692266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:03.692223    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj" podUID="9f057237-c5bb-4f1e-b8e3-ae085f4f1b92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:03:03.698943 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:03.698879    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6" podUID="35af209c-d9d9-4821-a124-2183e18a84c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:03:03.767399 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:03.767365    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:03.909663 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:03.909629    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:04.085838 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.085740    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:04.235077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.235027    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:04.593105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.593070    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:04.829571 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.829540    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 20:03:04.891185 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.891144    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 20:03:04.954396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.954361    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-kube-api-access-h82pb\" (UniqueName: \"kubernetes.io/projected/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-kube-api-access-h82pb\") pod \"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92\" (UID: \"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92\") "
Nov 03 20:03:04.954612 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.954595    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-plugins-config-volume\") pod \"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92\" (UID: \"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92\") "
Nov 03 20:03:04.955053 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.955022    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-plugins-config-volume" pod "9f057237-c5bb-4f1e-b8e3-ae085f4f1b92" (UID: "9f057237-c5bb-4f1e-b8e3-ae085f4f1b92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:04.955721 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.955696    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:04.956912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.956889    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-kube-api-access-h82pb" pod "9f057237-c5bb-4f1e-b8e3-ae085f4f1b92" (UID: "9f057237-c5bb-4f1e-b8e3-ae085f4f1b92"). InnerVolumeSpecName "kube-api-access-h82pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:04.995686 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.995653    4531 generic.go:356] "Generic (PLEG): container finished" podID="35af209c-d9d9-4821-a124-2183e18a84c5" containerID="8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0" exitCode=0
Nov 03 20:03:04.995835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.995702    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"
Nov 03 20:03:04.995835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.995732    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6" event={"ID":"35af209c-d9d9-4821-a124-2183e18a84c5","Type":"ContainerDied","Data":"8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0"}
Nov 03 20:03:04.995835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.995791    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6" event={"ID":"35af209c-d9d9-4821-a124-2183e18a84c5","Type":"ContainerDied","Data":"b953f9603618e762073aa97623262a9a419213fd01529ed93821883b2957a809"}
Nov 03 20:03:04.995835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.995822    4531 scope.go:122] "RemoveContainer" containerID="8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0"
Nov 03 20:03:04.997652 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.997628    4531 generic.go:356] "Generic (PLEG): container finished" podID="9f057237-c5bb-4f1e-b8e3-ae085f4f1b92" containerID="0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa" exitCode=0
Nov 03 20:03:04.997786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.997691    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"
Nov 03 20:03:04.997786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.997705    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj" event={"ID":"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92","Type":"ContainerDied","Data":"0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa"}
Nov 03 20:03:04.997786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:04.997745    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj" event={"ID":"9f057237-c5bb-4f1e-b8e3-ae085f4f1b92","Type":"ContainerDied","Data":"57429b1d37fd954553d938dbac74fb40296350ed38e7e38c2787b48c9b6052ed"}
Nov 03 20:03:05.010999 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.010977    4531 scope.go:122] "RemoveContainer" containerID="8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0"
Nov 03 20:03:05.011342 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:03:05.011319    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0\": container with ID starting with 8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0 not found: ID does not exist" containerID="8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0"
Nov 03 20:03:05.011404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.011352    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0"} err="failed to get container status \"8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0\": rpc error: code = NotFound desc = could not find container \"8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0\": container with ID starting with 8752f98ca4e979b47f65da64ce26ee9976cf786587ff21a0db1b80b1c51abde0 not found: ID does not exist"
Nov 03 20:03:05.011404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.011370    4531 scope.go:122] "RemoveContainer" containerID="0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa"
Nov 03 20:03:05.025435 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.025371    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"]
Nov 03 20:03:05.026312 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.026283    4531 scope.go:122] "RemoveContainer" containerID="0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa"
Nov 03 20:03:05.026667 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:03:05.026632    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa\": container with ID starting with 0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa not found: ID does not exist" containerID="0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa"
Nov 03 20:03:05.026774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.026673    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa"} err="failed to get container status \"0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa\": rpc error: code = NotFound desc = could not find container \"0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa\": container with ID starting with 0512ca1ccf709e51774038b042026f7a68b976d1ed40d0f29f1c130b2a2429fa not found: ID does not exist"
Nov 03 20:03:05.030146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.030123    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a99fad-58ff94c9ff-tjdqj"]
Nov 03 20:03:05.057159 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.057125    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35af209c-d9d9-4821-a124-2183e18a84c5-kube-api-access-zrh88\" (UniqueName: \"kubernetes.io/projected/35af209c-d9d9-4821-a124-2183e18a84c5-kube-api-access-zrh88\") pod \"35af209c-d9d9-4821-a124-2183e18a84c5\" (UID: \"35af209c-d9d9-4821-a124-2183e18a84c5\") "
Nov 03 20:03:05.057398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.057384    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35af209c-d9d9-4821-a124-2183e18a84c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35af209c-d9d9-4821-a124-2183e18a84c5-plugins-config-volume\") pod \"35af209c-d9d9-4821-a124-2183e18a84c5\" (UID: \"35af209c-d9d9-4821-a124-2183e18a84c5\") "
Nov 03 20:03:05.057731 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.057708    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35af209c-d9d9-4821-a124-2183e18a84c5-plugins-config-volume" pod "35af209c-d9d9-4821-a124-2183e18a84c5" (UID: "35af209c-d9d9-4821-a124-2183e18a84c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:05.057964 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.057947    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35af209c-d9d9-4821-a124-2183e18a84c5-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.058020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.057967    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h82pb\" (UniqueName: \"kubernetes.io/projected/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92-kube-api-access-h82pb\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.059361 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.059339    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35af209c-d9d9-4821-a124-2183e18a84c5-kube-api-access-zrh88" pod "35af209c-d9d9-4821-a124-2183e18a84c5" (UID: "35af209c-d9d9-4821-a124-2183e18a84c5"). InnerVolumeSpecName "kube-api-access-zrh88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:05.159581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.159541    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrh88\" (UniqueName: \"kubernetes.io/projected/35af209c-d9d9-4821-a124-2183e18a84c5-kube-api-access-zrh88\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.363290 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.354765    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:05.381527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.378432    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"]
Nov 03 20:03:05.389919 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.388781    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a99fad-5488bf9fdc-krzn6"]
Nov 03 20:03:05.429715 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.429678    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"]
Nov 03 20:03:05.439993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.439964    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"]
Nov 03 20:03:05.550326 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.550291    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:05.678675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:05.678634    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:06.380590 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:06.380556    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:06.862044 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:06.861946    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35af209c-d9d9-4821-a124-2183e18a84c5" path="/var/lib/kubelet/pods/35af209c-d9d9-4821-a124-2183e18a84c5/volumes"
Nov 03 20:03:06.862676 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:06.862650    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f057237-c5bb-4f1e-b8e3-ae085f4f1b92" path="/var/lib/kubelet/pods/9f057237-c5bb-4f1e-b8e3-ae085f4f1b92/volumes"
Nov 03 20:03:06.977122 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:06.977088    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:07.131895 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:07.131855    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:07.315450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:07.315415    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:07.929398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:07.929354    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:08.719471 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:08.717344    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:09.519653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:09.517305    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:10.320128 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.320089    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:10.430827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.430779    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws" podUID="958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb" containerName="payload-processing" containerID="cri-o://8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633" gracePeriod=25
Nov 03 20:03:10.440634 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.440596    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg" podUID="c9d1da86-ac7c-43eb-8381-142219ae2e9b" containerName="payload-pre-processing" containerID="cri-o://392997bd1370ed3e58aa0a575a72016e18f86f29703bb39eacb33300352a77e8" gracePeriod=25
Nov 03 20:03:10.682635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.682598    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:10.799284 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.799246    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 20:03:10.824873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.824836    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:10.854850 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.854809    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-kube-api-access-b98j2\" (UniqueName: \"kubernetes.io/projected/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-kube-api-access-b98j2\") pod \"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb\" (UID: \"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb\") "
Nov 03 20:03:10.855105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.855079    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-plugins-config-volume\") pod \"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb\" (UID: \"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb\") "
Nov 03 20:03:10.855599 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.855562    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-plugins-config-volume" pod "958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb" (UID: "958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:10.856267 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.855891    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:10.857620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.857596    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-kube-api-access-b98j2" pod "958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb" (UID: "958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb"). InnerVolumeSpecName "kube-api-access-b98j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:10.965234 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.958795    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b98j2\" (UniqueName: \"kubernetes.io/projected/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb-kube-api-access-b98j2\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:10.991195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.985824    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"]
Nov 03 20:03:10.991195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.986145    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy" containerID="cri-o://e9dd083a1b59d4f7975ff7ccaf73646ca402812d6ca9ebdb64b6d28d91157b33" gracePeriod=30
Nov 03 20:03:10.995574 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.992427    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"]
Nov 03 20:03:10.995574 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:10.992679    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy" containerID="cri-o://d5755f40c9b00101a1b050a2c8096b4df71b0a0ea4726d0cb606f14f45fc28d3" gracePeriod=30
Nov 03 20:03:11.060207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.060152    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:11.118985 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.118943    4531 generic.go:356] "Generic (PLEG): container finished" podID="958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb" containerID="8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633" exitCode=0
Nov 03 20:03:11.119187 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.118994    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"
Nov 03 20:03:11.119187 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.119006    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws" event={"ID":"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb","Type":"ContainerDied","Data":"8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633"}
Nov 03 20:03:11.119187 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.119072    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws" event={"ID":"958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb","Type":"ContainerDied","Data":"963d1c2bc8dcb58842f9ecf0377b4d36be9df05aa26379804fd70110708507d7"}
Nov 03 20:03:11.119187 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.119105    4531 scope.go:122] "RemoveContainer" containerID="8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633"
Nov 03 20:03:11.131703 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.131673    4531 scope.go:122] "RemoveContainer" containerID="8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633"
Nov 03 20:03:11.132069 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:03:11.132047    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633\": container with ID starting with 8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633 not found: ID does not exist" containerID="8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633"
Nov 03 20:03:11.132143 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.132081    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633"} err="failed to get container status \"8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633\": rpc error: code = NotFound desc = could not find container \"8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633\": container with ID starting with 8559866f74eebe8716ec72a875900d319a051e4f04abbf725d10b1afbd242633 not found: ID does not exist"
Nov 03 20:03:11.138992 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.138911    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"]
Nov 03 20:03:11.144015 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.143983    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-79c47f-746754fdd7-bnqws"]
Nov 03 20:03:11.174120 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.174081    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:03:11.174313 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.174140    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:03:11.227200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.225840    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:11.506638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.506541    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:03:11.506638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.506607    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:03:11.940095 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:11.940056    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:12.622188 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:12.621767    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:12.862379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:12.862345    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb" path="/var/lib/kubelet/pods/958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb/volumes"
Nov 03 20:03:13.380263 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:13.380225    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:14.036709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:14.036676    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:14.741211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:14.740344    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:15.401417 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:15.401379    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:16.127391 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.127329    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:16.167881 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.167843    4531 generic.go:356] "Generic (PLEG): container finished" podID="cb786573-56fb-4acd-b100-5ac76af38892" containerID="e9dd083a1b59d4f7975ff7ccaf73646ca402812d6ca9ebdb64b6d28d91157b33" exitCode=0
Nov 03 20:03:16.168056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.167920    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" event={"ID":"cb786573-56fb-4acd-b100-5ac76af38892","Type":"ContainerDied","Data":"e9dd083a1b59d4f7975ff7ccaf73646ca402812d6ca9ebdb64b6d28d91157b33"}
Nov 03 20:03:16.170265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.170235    4531 generic.go:356] "Generic (PLEG): container finished" podID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerID="d5755f40c9b00101a1b050a2c8096b4df71b0a0ea4726d0cb606f14f45fc28d3" exitCode=0
Nov 03 20:03:16.170425 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.170272    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" event={"ID":"eaa36d9d-eeaf-4a63-99da-862d2b923b0f","Type":"ContainerDied","Data":"d5755f40c9b00101a1b050a2c8096b4df71b0a0ea4726d0cb606f14f45fc28d3"}
Nov 03 20:03:16.265998 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.265845    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:16.434692 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.434657    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 20:03:16.522769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.522685    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-data\") pod \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") "
Nov 03 20:03:16.522769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.522731    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-podinfo\") pod \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") "
Nov 03 20:03:16.522769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.522755    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-credential-socket\") pod \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") "
Nov 03 20:03:16.523056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.522797    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istiod-ca-cert\") pod \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") "
Nov 03 20:03:16.523056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.522841    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-certs\") pod \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") "
Nov 03 20:03:16.523056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.522940    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-socket\") pod \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") "
Nov 03 20:03:16.523056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.523000    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-data" pod "eaa36d9d-eeaf-4a63-99da-862d2b923b0f" (UID: "eaa36d9d-eeaf-4a63-99da-862d2b923b0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.523056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.523045    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-envoy\") pod \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") "
Nov 03 20:03:16.523056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.523051    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-credential-socket" pod "eaa36d9d-eeaf-4a63-99da-862d2b923b0f" (UID: "eaa36d9d-eeaf-4a63-99da-862d2b923b0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.523407 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.523103    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-token\" (UniqueName: \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-token\") pod \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") "
Nov 03 20:03:16.523407 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.523234    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-socket" pod "eaa36d9d-eeaf-4a63-99da-862d2b923b0f" (UID: "eaa36d9d-eeaf-4a63-99da-862d2b923b0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.523407 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.523291    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-kube-api-access-mq6q4\" (UniqueName: \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-kube-api-access-mq6q4\") pod \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\" (UID: \"eaa36d9d-eeaf-4a63-99da-862d2b923b0f\") "
Nov 03 20:03:16.523581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.523470    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-certs" pod "eaa36d9d-eeaf-4a63-99da-862d2b923b0f" (UID: "eaa36d9d-eeaf-4a63-99da-862d2b923b0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.523652 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.523627    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istiod-ca-cert" pod "eaa36d9d-eeaf-4a63-99da-862d2b923b0f" (UID: "eaa36d9d-eeaf-4a63-99da-862d2b923b0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:16.524281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.524250    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.524402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.524284    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.524402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.524300    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.524402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.524352    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.524402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.524367    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.525913 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.525882    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-envoy" pod "eaa36d9d-eeaf-4a63-99da-862d2b923b0f" (UID: "eaa36d9d-eeaf-4a63-99da-862d2b923b0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.526735 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.526701    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-token" pod "eaa36d9d-eeaf-4a63-99da-862d2b923b0f" (UID: "eaa36d9d-eeaf-4a63-99da-862d2b923b0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:16.528376 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.528349    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-kube-api-access-mq6q4" pod "eaa36d9d-eeaf-4a63-99da-862d2b923b0f" (UID: "eaa36d9d-eeaf-4a63-99da-862d2b923b0f"). InnerVolumeSpecName "kube-api-access-mq6q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:16.530698 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.530275    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-podinfo" pod "eaa36d9d-eeaf-4a63-99da-862d2b923b0f" (UID: "eaa36d9d-eeaf-4a63-99da-862d2b923b0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:03:16.626609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.626572    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.626609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.626611    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.626928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.626625    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.626928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.626639    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq6q4\" (UniqueName: \"kubernetes.io/projected/eaa36d9d-eeaf-4a63-99da-862d2b923b0f-kube-api-access-mq6q4\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.814492 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.814457    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 20:03:16.926309 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.926277    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:16.936549 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.936515    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb786573-56fb-4acd-b100-5ac76af38892-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb786573-56fb-4acd-b100-5ac76af38892-istiod-ca-cert\") pod \"cb786573-56fb-4acd-b100-5ac76af38892\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") "
Nov 03 20:03:16.936709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.936612    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb786573-56fb-4acd-b100-5ac76af38892-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb786573-56fb-4acd-b100-5ac76af38892-istio-podinfo\") pod \"cb786573-56fb-4acd-b100-5ac76af38892\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") "
Nov 03 20:03:16.936709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.936637    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-envoy\") pod \"cb786573-56fb-4acd-b100-5ac76af38892\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") "
Nov 03 20:03:16.936709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.936656    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-istio-token\" (UniqueName: \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-istio-token\") pod \"cb786573-56fb-4acd-b100-5ac76af38892\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") "
Nov 03 20:03:16.936900 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.936811    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-credential-socket\") pod \"cb786573-56fb-4acd-b100-5ac76af38892\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") "
Nov 03 20:03:16.936963 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.936938    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb786573-56fb-4acd-b100-5ac76af38892-istiod-ca-cert" pod "cb786573-56fb-4acd-b100-5ac76af38892" (UID: "cb786573-56fb-4acd-b100-5ac76af38892"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:16.937020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.936985    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-certs\") pod \"cb786573-56fb-4acd-b100-5ac76af38892\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") "
Nov 03 20:03:16.937072 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.937021    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-kube-api-access-clcgl\" (UniqueName: \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-kube-api-access-clcgl\") pod \"cb786573-56fb-4acd-b100-5ac76af38892\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") "
Nov 03 20:03:16.937126 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.937081    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-data\") pod \"cb786573-56fb-4acd-b100-5ac76af38892\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") "
Nov 03 20:03:16.937211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.937125    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-credential-socket" pod "cb786573-56fb-4acd-b100-5ac76af38892" (UID: "cb786573-56fb-4acd-b100-5ac76af38892"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.937211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.937138    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-socket\") pod \"cb786573-56fb-4acd-b100-5ac76af38892\" (UID: \"cb786573-56fb-4acd-b100-5ac76af38892\") "
Nov 03 20:03:16.937364 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.937301    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-certs" pod "cb786573-56fb-4acd-b100-5ac76af38892" (UID: "cb786573-56fb-4acd-b100-5ac76af38892"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.937429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.937398    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-data" pod "cb786573-56fb-4acd-b100-5ac76af38892" (UID: "cb786573-56fb-4acd-b100-5ac76af38892"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.937589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.937546    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-socket" pod "cb786573-56fb-4acd-b100-5ac76af38892" (UID: "cb786573-56fb-4acd-b100-5ac76af38892"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.939052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.938252    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.939052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.938283    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.939052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.938299    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.939052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.938312    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb786573-56fb-4acd-b100-5ac76af38892-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.939052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.938326    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.939435 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.939373    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb786573-56fb-4acd-b100-5ac76af38892-istio-podinfo" pod "cb786573-56fb-4acd-b100-5ac76af38892" (UID: "cb786573-56fb-4acd-b100-5ac76af38892"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:03:16.939636 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.939613    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-envoy" pod "cb786573-56fb-4acd-b100-5ac76af38892" (UID: "cb786573-56fb-4acd-b100-5ac76af38892"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.939742 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.939654    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-kube-api-access-clcgl" pod "cb786573-56fb-4acd-b100-5ac76af38892" (UID: "cb786573-56fb-4acd-b100-5ac76af38892"). InnerVolumeSpecName "kube-api-access-clcgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:16.939742 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:16.939681    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-istio-token" pod "cb786573-56fb-4acd-b100-5ac76af38892" (UID: "cb786573-56fb-4acd-b100-5ac76af38892"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:17.040295 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.040259    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clcgl\" (UniqueName: \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-kube-api-access-clcgl\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:17.040457 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.040297    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb786573-56fb-4acd-b100-5ac76af38892-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:17.040457 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.040314    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb786573-56fb-4acd-b100-5ac76af38892-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:17.040457 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.040327    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb786573-56fb-4acd-b100-5ac76af38892-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:17.243285 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.240761    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8" event={"ID":"eaa36d9d-eeaf-4a63-99da-862d2b923b0f","Type":"ContainerDied","Data":"4c404812a5a231a76d7fa7365dde64c9580d9b24f1eaf618784d0aa96dcee195"}
Nov 03 20:03:17.243285 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.240841    4531 scope.go:122] "RemoveContainer" containerID="d5755f40c9b00101a1b050a2c8096b4df71b0a0ea4726d0cb606f14f45fc28d3"
Nov 03 20:03:17.243285 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.241002    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"
Nov 03 20:03:17.276762 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.276721    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs" event={"ID":"cb786573-56fb-4acd-b100-5ac76af38892","Type":"ContainerDied","Data":"ff70392365b959e1fc7f7054e63c899ef53ef14463ced7e8d9ff44d01f50712e"}
Nov 03 20:03:17.278093 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.277920    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"
Nov 03 20:03:17.379798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.379265    4531 scope.go:122] "RemoveContainer" containerID="e9dd083a1b59d4f7975ff7ccaf73646ca402812d6ca9ebdb64b6d28d91157b33"
Nov 03 20:03:17.379798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.379425    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"]
Nov 03 20:03:17.393003 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.389921    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-v74q8"]
Nov 03 20:03:17.410510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.410429    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"]
Nov 03 20:03:17.416722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.416692    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-fvfxs"]
Nov 03 20:03:17.542867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.542751    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:17.657351 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.656583    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"]
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.657900    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.657926    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.657948    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b0235d15-197e-47e9-b122-e6e49eefbe9f" containerName="revoke-keys"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.657955    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0235d15-197e-47e9-b122-e6e49eefbe9f" containerName="revoke-keys"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.657972    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.657979    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.657990    4531 cpu_manager.go:414] "containerMap: removing container" podUID="35af209c-d9d9-4821-a124-2183e18a84c5" containerName="payload-pre-processing"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658000    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35af209c-d9d9-4821-a124-2183e18a84c5" containerName="payload-pre-processing"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658014    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658021    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658031    4531 cpu_manager.go:414] "containerMap: removing container" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658038    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658070    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9f057237-c5bb-4f1e-b8e3-ae085f4f1b92" containerName="payload-processing"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658077    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f057237-c5bb-4f1e-b8e3-ae085f4f1b92" containerName="payload-processing"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658104    4531 cpu_manager.go:414] "containerMap: removing container" podUID="74560e30-953f-48f5-8031-9a73b7218cef" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658111    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74560e30-953f-48f5-8031-9a73b7218cef" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658144    4531 cpu_manager.go:414] "containerMap: removing container" podUID="958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb" containerName="payload-processing"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658151    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb" containerName="payload-processing"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658363    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f057237-c5bb-4f1e-b8e3-ae085f4f1b92" containerName="payload-processing"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658385    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="74560e30-953f-48f5-8031-9a73b7218cef" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658400    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="958e2cae-1ff0-4e97-8bd9-a9fbdac37dcb" containerName="payload-processing"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658415    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb786573-56fb-4acd-b100-5ac76af38892" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658426    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0235d15-197e-47e9-b122-e6e49eefbe9f" containerName="revoke-keys"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658434    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c80951b-0a70-422b-acab-00b767b5304f" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658444    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="35af209c-d9d9-4821-a124-2183e18a84c5" containerName="payload-pre-processing"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658453    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9883aa-fca8-416e-8667-85354d4c2702" containerName="istio-proxy"
Nov 03 20:03:17.660357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.658464    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" containerName="istio-proxy"
Nov 03 20:03:17.664599 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.664576    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"]
Nov 03 20:03:17.664707 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.664667    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:17.668592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.667490    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:03:17.668592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.667602    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:03:17.780374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.780333    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55674fc2-jj76l\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:17.780559 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.780536    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bpf9\" (UniqueName: \"kubernetes.io/projected/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-kube-api-access-5bpf9\") pod \"maas-api-revoke-sub-55674fc2-jj76l\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:17.780673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.780650    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55674fc2-jj76l\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:17.882092 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.882046    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55674fc2-jj76l\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:17.882345 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.882324    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bpf9\" (UniqueName: \"kubernetes.io/projected/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-kube-api-access-5bpf9\") pod \"maas-api-revoke-sub-55674fc2-jj76l\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:17.882459 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.882439    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55674fc2-jj76l\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:17.883325 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.883298    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55674fc2-jj76l\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:17.884757 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.884740    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55674fc2-jj76l\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:17.890318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.890286    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bpf9\" (UniqueName: \"kubernetes.io/projected/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-kube-api-access-5bpf9\") pod \"maas-api-revoke-sub-55674fc2-jj76l\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:17.982654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:17.982619    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:18.266028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:18.266002    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"]
Nov 03 20:03:18.266616 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:03:18.266589    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ecbfff_be37_40a2_90cb_bf95ab9c5a28.slice/crio-5069e26157798913562e7bd339f3a1171951d864cb687331504520542f92dc43 WatchSource:0}: Error finding container 5069e26157798913562e7bd339f3a1171951d864cb687331504520542f92dc43: Status 404 returned error can't find the container with id 5069e26157798913562e7bd339f3a1171951d864cb687331504520542f92dc43
Nov 03 20:03:18.327393 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:18.327344    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:18.336322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:18.332638    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l" event={"ID":"38ecbfff-be37-40a2-90cb-bf95ab9c5a28","Type":"ContainerStarted","Data":"5069e26157798913562e7bd339f3a1171951d864cb687331504520542f92dc43"}
Nov 03 20:03:18.861846 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:18.861768    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb786573-56fb-4acd-b100-5ac76af38892" path="/var/lib/kubelet/pods/cb786573-56fb-4acd-b100-5ac76af38892/volumes"
Nov 03 20:03:18.862369 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:18.862352    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa36d9d-eeaf-4a63-99da-862d2b923b0f" path="/var/lib/kubelet/pods/eaa36d9d-eeaf-4a63-99da-862d2b923b0f/volumes"
Nov 03 20:03:18.879751 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:18.879721    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:19.051194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:19.050639    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:19.205195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:19.197627    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:19.236303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:19.235939    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"]
Nov 03 20:03:19.267265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:19.267148    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-79c47f-2cnnb"]
Nov 03 20:03:19.371609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:19.365763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:19.393250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:19.391622    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l" event={"ID":"38ecbfff-be37-40a2-90cb-bf95ab9c5a28","Type":"ContainerStarted","Data":"c96f36b6e713c2e0a1d48a8450037894d8e0fd97e35354d8676f0d6c1e4da0a4"}
Nov 03 20:03:19.410979 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:19.410918    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l" podStartSLOduration=2.410903271 podStartE2EDuration="2.410903271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:19.406287975 +0000 UTC m=+1912.676411270" watchObservedRunningTime="2026-11-03 20:03:19.410903271 +0000 UTC m=+1912.681026617"
Nov 03 20:03:19.609073 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:19.608982    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:19.975352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:19.975311    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg" podUID="c9d1da86-ac7c-43eb-8381-142219ae2e9b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:03:20.374869 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:20.371591    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:20.411817 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:20.411780    4531 generic.go:356] "Generic (PLEG): container finished" podID="38ecbfff-be37-40a2-90cb-bf95ab9c5a28" containerID="c96f36b6e713c2e0a1d48a8450037894d8e0fd97e35354d8676f0d6c1e4da0a4" exitCode=0
Nov 03 20:03:20.412116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:20.412091    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l" event={"ID":"38ecbfff-be37-40a2-90cb-bf95ab9c5a28","Type":"ContainerDied","Data":"c96f36b6e713c2e0a1d48a8450037894d8e0fd97e35354d8676f0d6c1e4da0a4"}
Nov 03 20:03:20.575908 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:03:20.575865    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:03:20.860682 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:20.860646    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0235d15-197e-47e9-b122-e6e49eefbe9f" path="/var/lib/kubelet/pods/b0235d15-197e-47e9-b122-e6e49eefbe9f/volumes"
Nov 03 20:03:21.242242 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:21.241797    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:21.598250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:21.598216    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:21.743081 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:21.742934    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-openshift-ca\") pod \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") "
Nov 03 20:03:21.743308 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:21.743167    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-xks-ca\") pod \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") "
Nov 03 20:03:21.743308 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:21.743261    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-kube-api-access-5bpf9\" (UniqueName: \"kubernetes.io/projected/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-kube-api-access-5bpf9\") pod \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\" (UID: \"38ecbfff-be37-40a2-90cb-bf95ab9c5a28\") "
Nov 03 20:03:21.743450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:21.743414    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-openshift-ca" pod "38ecbfff-be37-40a2-90cb-bf95ab9c5a28" (UID: "38ecbfff-be37-40a2-90cb-bf95ab9c5a28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:21.744376 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:21.744346    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:21.746565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:21.746387    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-xks-ca" pod "38ecbfff-be37-40a2-90cb-bf95ab9c5a28" (UID: "38ecbfff-be37-40a2-90cb-bf95ab9c5a28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:03:21.746565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:21.746433    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-kube-api-access-5bpf9" pod "38ecbfff-be37-40a2-90cb-bf95ab9c5a28" (UID: "38ecbfff-be37-40a2-90cb-bf95ab9c5a28"). InnerVolumeSpecName "kube-api-access-5bpf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:21.846573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:21.846469    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:21.846573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:21.846505    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bpf9\" (UniqueName: \"kubernetes.io/projected/38ecbfff-be37-40a2-90cb-bf95ab9c5a28-kube-api-access-5bpf9\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:22.041577 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.041544    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:22.193329 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.187959    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:22.432394 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.432365    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"
Nov 03 20:03:22.432573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.432387    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l" event={"ID":"38ecbfff-be37-40a2-90cb-bf95ab9c5a28","Type":"ContainerDied","Data":"5069e26157798913562e7bd339f3a1171951d864cb687331504520542f92dc43"}
Nov 03 20:03:22.432573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.432438    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5069e26157798913562e7bd339f3a1171951d864cb687331504520542f92dc43"
Nov 03 20:03:22.706270 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.706230    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"]
Nov 03 20:03:22.721198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.708206    4531 cpu_manager.go:414] "containerMap: removing container" podUID="38ecbfff-be37-40a2-90cb-bf95ab9c5a28" containerName="revoke-keys"
Nov 03 20:03:22.721556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.721515    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ecbfff-be37-40a2-90cb-bf95ab9c5a28" containerName="revoke-keys"
Nov 03 20:03:22.721968 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.721950    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ecbfff-be37-40a2-90cb-bf95ab9c5a28" containerName="revoke-keys"
Nov 03 20:03:22.736486 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.736454    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.743349 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.742124    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"]
Nov 03 20:03:22.749035 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.749006    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f58171-openshift-default-dockercfg-kj42z\""
Nov 03 20:03:22.774672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.774631    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-data\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.774861 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.774761    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24b4233-7e81-47f5-90b3-74601c85022b-istio-podinfo\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.774861 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.774830    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.774989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.774902    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24b4233-7e81-47f5-90b3-74601c85022b-istiod-ca-cert\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.774989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.774965    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-envoy\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.775107 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.775001    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-certs\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.775168 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.775121    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phqb4\" (UniqueName: \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-kube-api-access-phqb4\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.775168 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.775139    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-istio-token\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.775314 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.775187    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-credential-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.876940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.876905    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24b4233-7e81-47f5-90b3-74601c85022b-istio-podinfo\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.876940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.876941    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.877221 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.876982    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24b4233-7e81-47f5-90b3-74601c85022b-istiod-ca-cert\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.877221 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.877008    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-envoy\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.877221 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.877034    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-certs\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.877221 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.877132    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phqb4\" (UniqueName: \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-kube-api-access-phqb4\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.877221 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.877160    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-istio-token\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.877221 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.877221    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-credential-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.877566 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.877464    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-data\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.877566 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.877479    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.877778 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.877754    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-data\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.877862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.877817    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24b4233-7e81-47f5-90b3-74601c85022b-istiod-ca-cert\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.877862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.877828    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-credential-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.878023 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.878001    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-certs\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.879524 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.879497    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-envoy\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.879769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.879748    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24b4233-7e81-47f5-90b3-74601c85022b-istio-podinfo\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.884970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.884942    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phqb4\" (UniqueName: \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-kube-api-access-phqb4\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:22.887467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:22.887447    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-istio-token\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:23.052342 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:23.052248    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:23.064492 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:23.064443    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:23.629673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:23.629631    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"]
Nov 03 20:03:23.631076 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:03:23.631041    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24b4233_7e81_47f5_90b3_74601c85022b.slice/crio-d7c84ac34c006b74f51b4fede126657b770f1b9597329f822733768f4d3ace0c WatchSource:0}: Error finding container d7c84ac34c006b74f51b4fede126657b770f1b9597329f822733768f4d3ace0c: Status 404 returned error can't find the container with id d7c84ac34c006b74f51b4fede126657b770f1b9597329f822733768f4d3ace0c
Nov 03 20:03:23.634372 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:23.634329    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:23.634493 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:23.634416    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:23.634493 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:23.634445    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:23.993158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:23.993056    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:24.471865 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:24.471813    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" event={"ID":"a24b4233-7e81-47f5-90b3-74601c85022b","Type":"ContainerStarted","Data":"b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71"}
Nov 03 20:03:24.472086 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:24.471927    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" event={"ID":"a24b4233-7e81-47f5-90b3-74601c85022b","Type":"ContainerStarted","Data":"d7c84ac34c006b74f51b4fede126657b770f1b9597329f822733768f4d3ace0c"}
Nov 03 20:03:24.492107 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:24.492040    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" podStartSLOduration=2.492024239 podStartE2EDuration="2.492024239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:24.489488746 +0000 UTC m=+1917.759612042" watchObservedRunningTime="2026-11-03 20:03:24.492024239 +0000 UTC m=+1917.762147534"
Nov 03 20:03:24.836383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:24.836289    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:25.065575 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:25.065527    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:25.067604 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:25.067577    4531 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 20:03:25.067739 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:25.067643    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" podUID="a24b4233-7e81-47f5-90b3-74601c85022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 20:03:25.678199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:25.676304    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:26.066024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:26.065930    4531 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 20:03:26.066452 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:26.066004    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" podUID="a24b4233-7e81-47f5-90b3-74601c85022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 20:03:26.133198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:26.121243    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:26.262197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:26.262142    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:26.372907 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:26.372785    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"]
Nov 03 20:03:26.373423 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:26.373390    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy" containerID="cri-o://e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de" gracePeriod=30
Nov 03 20:03:26.412766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:26.412735    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:26.600198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:26.589933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:27.067201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:27.066227    4531 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 20:03:27.067201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:27.066312    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" podUID="a24b4233-7e81-47f5-90b3-74601c85022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 20:03:27.591204 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:27.574937    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:27.975636 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:27.975047    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"]
Nov 03 20:03:27.985776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:27.982987    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:27.985967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:27.985795    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"]
Nov 03 20:03:27.987818 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:27.987788    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9f3188\"/\"kube-root-ca.crt\""
Nov 03 20:03:27.991023 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:27.990998    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9f3188\"/\"default-dockercfg-bgpz8\""
Nov 03 20:03:27.991818 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:27.991274    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9f3188\"/\"openshift-service-ca.crt\""
Nov 03 20:03:27.991999 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:27.991915    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9f3188\"/\"auth-test-model-9f3188-kserve-self-signed-certs\""
Nov 03 20:03:28.065194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.064778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:28.065521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.065425    4531 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 20:03:28.065521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.065479    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" podUID="a24b4233-7e81-47f5-90b3-74601c85022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 20:03:28.118603 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.118557    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tmp-dir\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.138199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.120216    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-model-cache\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.138199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.120428    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngmhq\" (UniqueName: \"kubernetes.io/projected/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-kube-api-access-ngmhq\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.138199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.120468    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-home\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.138199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.120530    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-dshm\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.138199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.120725    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tls-certs\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.257254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.245760    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-model-cache\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.257254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.244782    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-model-cache\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.257254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.246446    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngmhq\" (UniqueName: \"kubernetes.io/projected/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-kube-api-access-ngmhq\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.257254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.246483    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-home\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.257254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.246550    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-dshm\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.257254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.246834    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tls-certs\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.257254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.248150    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-home\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.269197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.267479    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tmp-dir\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.269197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.246961    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tmp-dir\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.278383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.276624    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-dshm\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.278383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.276929    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tls-certs\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.296195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.288027    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngmhq\" (UniqueName: \"kubernetes.io/projected/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-kube-api-access-ngmhq\") pod \"auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") " pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.360230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.347725    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:28.471280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.463720    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:28.786131 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:03:28.786096    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fa5ca17_a405_43b0_bf42_acdf79ff1a70.slice/crio-21ab63c39b24cd0112f2796f4b3e550f7d979fcedfb8c41fee9637daa329feda WatchSource:0}: Error finding container 21ab63c39b24cd0112f2796f4b3e550f7d979fcedfb8c41fee9637daa329feda: Status 404 returned error can't find the container with id 21ab63c39b24cd0112f2796f4b3e550f7d979fcedfb8c41fee9637daa329feda
Nov 03 20:03:28.787099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:28.787067    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"]
Nov 03 20:03:29.065346 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:29.065306    4531 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 20:03:29.065535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:29.065374    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" podUID="a24b4233-7e81-47f5-90b3-74601c85022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 20:03:29.190016 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:29.189981    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:29.671332 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:29.671292    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p" event={"ID":"8fa5ca17-a405-43b0-bf42-acdf79ff1a70","Type":"ContainerStarted","Data":"009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508"}
Nov 03 20:03:29.671553 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:29.671344    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p" event={"ID":"8fa5ca17-a405-43b0-bf42-acdf79ff1a70","Type":"ContainerStarted","Data":"21ab63c39b24cd0112f2796f4b3e550f7d979fcedfb8c41fee9637daa329feda"}
Nov 03 20:03:29.689454 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:29.689395    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p" podStartSLOduration=2.68938 podStartE2EDuration="2.68938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:29.685717864 +0000 UTC m=+1922.955841157" watchObservedRunningTime="2026-11-03 20:03:29.68938 +0000 UTC m=+1922.959503294"
Nov 03 20:03:29.865200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:29.857735    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:29.975540 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:29.975448    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg" podUID="c9d1da86-ac7c-43eb-8381-142219ae2e9b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:03:30.065078 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.065037    4531 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 20:03:30.065302 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.065093    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" podUID="a24b4233-7e81-47f5-90b3-74601c85022b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 20:03:30.357245 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.357087    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:30.612268 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.607735    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:30.739877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.739834    4531 generic.go:356] "Generic (PLEG): container finished" podID="c9d1da86-ac7c-43eb-8381-142219ae2e9b" containerID="392997bd1370ed3e58aa0a575a72016e18f86f29703bb39eacb33300352a77e8" exitCode=0
Nov 03 20:03:30.740097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.739939    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg" event={"ID":"c9d1da86-ac7c-43eb-8381-142219ae2e9b","Type":"ContainerDied","Data":"392997bd1370ed3e58aa0a575a72016e18f86f29703bb39eacb33300352a77e8"}
Nov 03 20:03:30.833825 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.833797    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 20:03:30.916099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.916050    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d1da86-ac7c-43eb-8381-142219ae2e9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d1da86-ac7c-43eb-8381-142219ae2e9b-plugins-config-volume\") pod \"c9d1da86-ac7c-43eb-8381-142219ae2e9b\" (UID: \"c9d1da86-ac7c-43eb-8381-142219ae2e9b\") "
Nov 03 20:03:30.916341 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.916202    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d1da86-ac7c-43eb-8381-142219ae2e9b-kube-api-access-pp4zn\" (UniqueName: \"kubernetes.io/projected/c9d1da86-ac7c-43eb-8381-142219ae2e9b-kube-api-access-pp4zn\") pod \"c9d1da86-ac7c-43eb-8381-142219ae2e9b\" (UID: \"c9d1da86-ac7c-43eb-8381-142219ae2e9b\") "
Nov 03 20:03:30.916475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.916451    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d1da86-ac7c-43eb-8381-142219ae2e9b-plugins-config-volume" pod "c9d1da86-ac7c-43eb-8381-142219ae2e9b" (UID: "c9d1da86-ac7c-43eb-8381-142219ae2e9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:30.916883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.916860    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d1da86-ac7c-43eb-8381-142219ae2e9b-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:30.918497 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:30.918467    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d1da86-ac7c-43eb-8381-142219ae2e9b-kube-api-access-pp4zn" pod "c9d1da86-ac7c-43eb-8381-142219ae2e9b" (UID: "c9d1da86-ac7c-43eb-8381-142219ae2e9b"). InnerVolumeSpecName "kube-api-access-pp4zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:31.018104 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.018056    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp4zn\" (UniqueName: \"kubernetes.io/projected/c9d1da86-ac7c-43eb-8381-142219ae2e9b-kube-api-access-pp4zn\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:31.070885 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.070847    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:31.071241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.071216    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:31.072090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.072073    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:03:31.407312 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.407277    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:31.710937 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.710910    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 20:03:31.731835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.731794    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-kube-api-access-wkqsc\" (UniqueName: \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-kube-api-access-wkqsc\") pod \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") "
Nov 03 20:03:31.732014 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.731915    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-podinfo\") pod \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") "
Nov 03 20:03:31.732014 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.731958    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-token\" (UniqueName: \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-token\") pod \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") "
Nov 03 20:03:31.732160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.732012    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istiod-ca-cert\") pod \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") "
Nov 03 20:03:31.732160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.732073    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-envoy\") pod \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") "
Nov 03 20:03:31.732160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.732123    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-certs\") pod \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") "
Nov 03 20:03:31.732350 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.732217    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-data\") pod \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") "
Nov 03 20:03:31.732350 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.732252    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-socket\") pod \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") "
Nov 03 20:03:31.732350 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.732313    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-credential-socket\") pod \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\" (UID: \"bd6ef0ea-9769-4121-8f44-39adf7de2e3b\") "
Nov 03 20:03:31.732565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.732534    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-certs" pod "bd6ef0ea-9769-4121-8f44-39adf7de2e3b" (UID: "bd6ef0ea-9769-4121-8f44-39adf7de2e3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:31.732631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.732566    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-data" pod "bd6ef0ea-9769-4121-8f44-39adf7de2e3b" (UID: "bd6ef0ea-9769-4121-8f44-39adf7de2e3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:31.732631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.732600    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istiod-ca-cert" pod "bd6ef0ea-9769-4121-8f44-39adf7de2e3b" (UID: "bd6ef0ea-9769-4121-8f44-39adf7de2e3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:31.733100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.732880    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-credential-socket" pod "bd6ef0ea-9769-4121-8f44-39adf7de2e3b" (UID: "bd6ef0ea-9769-4121-8f44-39adf7de2e3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:31.733244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.733112    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-socket" pod "bd6ef0ea-9769-4121-8f44-39adf7de2e3b" (UID: "bd6ef0ea-9769-4121-8f44-39adf7de2e3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:31.733681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.733577    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:31.733681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.733608    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:31.733681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.733625    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:31.733681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.733641    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:31.733681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.733657    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:31.735249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.735221    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-envoy" pod "bd6ef0ea-9769-4121-8f44-39adf7de2e3b" (UID: "bd6ef0ea-9769-4121-8f44-39adf7de2e3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:31.735429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.735399    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-token" pod "bd6ef0ea-9769-4121-8f44-39adf7de2e3b" (UID: "bd6ef0ea-9769-4121-8f44-39adf7de2e3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:31.735853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.735828    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-podinfo" pod "bd6ef0ea-9769-4121-8f44-39adf7de2e3b" (UID: "bd6ef0ea-9769-4121-8f44-39adf7de2e3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:03:31.736543 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.736525    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-kube-api-access-wkqsc" pod "bd6ef0ea-9769-4121-8f44-39adf7de2e3b" (UID: "bd6ef0ea-9769-4121-8f44-39adf7de2e3b"). InnerVolumeSpecName "kube-api-access-wkqsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:31.755613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.755555    4531 generic.go:356] "Generic (PLEG): container finished" podID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerID="e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de" exitCode=0
Nov 03 20:03:31.755976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.755936    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" event={"ID":"bd6ef0ea-9769-4121-8f44-39adf7de2e3b","Type":"ContainerDied","Data":"e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de"}
Nov 03 20:03:31.756095 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.756044    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj" event={"ID":"bd6ef0ea-9769-4121-8f44-39adf7de2e3b","Type":"ContainerDied","Data":"e337cc1d0d4b949d97b791b05ecb74e6aef896af80d363521a347cb674e3b3ad"}
Nov 03 20:03:31.756095 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.756076    4531 scope.go:122] "RemoveContainer" containerID="e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de"
Nov 03 20:03:31.756346 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.756266    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"
Nov 03 20:03:31.765667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.765634    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg" event={"ID":"c9d1da86-ac7c-43eb-8381-142219ae2e9b","Type":"ContainerDied","Data":"6e1b91839ea890572f04e0cff1941201b176ba032121f0d547f5f6492c63b2bf"}
Nov 03 20:03:31.766035 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.765998    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"
Nov 03 20:03:31.783781 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.783742    4531 scope.go:122] "RemoveContainer" containerID="3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614"
Nov 03 20:03:31.794384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.794351    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"]
Nov 03 20:03:31.800970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.800943    4531 scope.go:122] "RemoveContainer" containerID="e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de"
Nov 03 20:03:31.801357 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:03:31.801335    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de\": container with ID starting with e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de not found: ID does not exist" containerID="e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de"
Nov 03 20:03:31.801473 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.801369    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de"} err="failed to get container status \"e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de\": rpc error: code = NotFound desc = could not find container \"e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de\": container with ID starting with e21900378cb5a87d6112cddd29e57e262e2ace1c2e75c526ccfac7fe70cbf7de not found: ID does not exist"
Nov 03 20:03:31.801473 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.801394    4531 scope.go:122] "RemoveContainer" containerID="3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614"
Nov 03 20:03:31.801743 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:03:31.801718    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614\": container with ID starting with 3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614 not found: ID does not exist" containerID="3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614"
Nov 03 20:03:31.801803 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.801752    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614"} err="failed to get container status \"3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614\": rpc error: code = NotFound desc = could not find container \"3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614\": container with ID starting with 3e527d7b2d0e0b3e3ec7b9994769721afad9b37ad773a6a07b8141b30d885614 not found: ID does not exist"
Nov 03 20:03:31.801803 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.801786    4531 scope.go:122] "RemoveContainer" containerID="392997bd1370ed3e58aa0a575a72016e18f86f29703bb39eacb33300352a77e8"
Nov 03 20:03:31.806460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.806438    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79c47f-b44bd7475-6zngg"]
Nov 03 20:03:31.815828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.815805    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"]
Nov 03 20:03:31.820866 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.820844    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-fcvnj"]
Nov 03 20:03:31.835912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.835890    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkqsc\" (UniqueName: \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-kube-api-access-wkqsc\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:31.835912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.835911    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:31.836092 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.835921    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:31.836092 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:31.835930    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6ef0ea-9769-4121-8f44-39adf7de2e3b-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:32.088047 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:32.087961    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:32.887909 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:32.887877    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:32.898690 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:32.898653    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" path="/var/lib/kubelet/pods/bd6ef0ea-9769-4121-8f44-39adf7de2e3b/volumes"
Nov 03 20:03:32.899688 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:32.899657    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d1da86-ac7c-43eb-8381-142219ae2e9b" path="/var/lib/kubelet/pods/c9d1da86-ac7c-43eb-8381-142219ae2e9b/volumes"
Nov 03 20:03:33.639604 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:33.639534    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:33.848201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:33.845273    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:34.544543 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:34.539009    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:35.358383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:35.354495    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:36.019295 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.015593    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:36.277124 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.277029    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:36.696621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.696582    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"]
Nov 03 20:03:36.697884 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.697851    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy"
Nov 03 20:03:36.697884 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.697880    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy"
Nov 03 20:03:36.698061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.697895    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy"
Nov 03 20:03:36.698061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.697903    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy"
Nov 03 20:03:36.698061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.697918    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c9d1da86-ac7c-43eb-8381-142219ae2e9b" containerName="payload-pre-processing"
Nov 03 20:03:36.698061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.697951    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d1da86-ac7c-43eb-8381-142219ae2e9b" containerName="payload-pre-processing"
Nov 03 20:03:36.698309 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.698139    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy"
Nov 03 20:03:36.698309 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.698152    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6ef0ea-9769-4121-8f44-39adf7de2e3b" containerName="istio-proxy"
Nov 03 20:03:36.698309 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.698164    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d1da86-ac7c-43eb-8381-142219ae2e9b" containerName="payload-pre-processing"
Nov 03 20:03:36.711387 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.711356    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"]
Nov 03 20:03:36.711570 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.711476    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:36.714958 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.714930    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:03:36.715113 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.714999    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:03:36.804210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.804149    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a48046ae-nbts4\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:36.804457 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.804422    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a48046ae-nbts4\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:36.804525 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.804482    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8j6t\" (UniqueName: \"kubernetes.io/projected/811cc1b2-8239-4b7f-ab01-08eba26baed8-kube-api-access-r8j6t\") pod \"maas-api-revoke-sub-a48046ae-nbts4\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:36.828517 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.828486    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:36.906058 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.906020    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a48046ae-nbts4\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:36.906340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.906099    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j6t\" (UniqueName: \"kubernetes.io/projected/811cc1b2-8239-4b7f-ab01-08eba26baed8-kube-api-access-r8j6t\") pod \"maas-api-revoke-sub-a48046ae-nbts4\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:36.906340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.906294    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a48046ae-nbts4\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:36.909031 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.909001    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a48046ae-nbts4\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:36.913505 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.913475    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a48046ae-nbts4\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:36.920472 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:36.920442    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8j6t\" (UniqueName: \"kubernetes.io/projected/811cc1b2-8239-4b7f-ab01-08eba26baed8-kube-api-access-r8j6t\") pod \"maas-api-revoke-sub-a48046ae-nbts4\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:37.037694 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:37.037603    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:37.325515 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:03:37.325482    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811cc1b2_8239_4b7f_ab01_08eba26baed8.slice/crio-a996985de77ce8b37862541dc8d9185186b4f9a8c6225736597a0ba9053a0b2d WatchSource:0}: Error finding container a996985de77ce8b37862541dc8d9185186b4f9a8c6225736597a0ba9053a0b2d: Status 404 returned error can't find the container with id a996985de77ce8b37862541dc8d9185186b4f9a8c6225736597a0ba9053a0b2d
Nov 03 20:03:37.326651 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:37.326623    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"]
Nov 03 20:03:37.851831 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:37.851740    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:37.947637 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:37.947593    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4" event={"ID":"811cc1b2-8239-4b7f-ab01-08eba26baed8","Type":"ContainerStarted","Data":"665d2783255fca8e1c89527b8f8b3d98f099743bade2e132df46894f964ab330"}
Nov 03 20:03:37.947834 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:37.947667    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4" event={"ID":"811cc1b2-8239-4b7f-ab01-08eba26baed8","Type":"ContainerStarted","Data":"a996985de77ce8b37862541dc8d9185186b4f9a8c6225736597a0ba9053a0b2d"}
Nov 03 20:03:37.963277 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:37.963195    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4" podStartSLOduration=1.9631621639999999 podStartE2EDuration="1.963162164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:37.961123488 +0000 UTC m=+1931.231246786" watchObservedRunningTime="2026-11-03 20:03:37.963162164 +0000 UTC m=+1931.233285458"
Nov 03 20:03:38.356411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:38.356365    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:38.370104 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:38.370055    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:38.702635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:38.701218    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:38.958626 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:38.958533    4531 generic.go:356] "Generic (PLEG): container finished" podID="811cc1b2-8239-4b7f-ab01-08eba26baed8" containerID="665d2783255fca8e1c89527b8f8b3d98f099743bade2e132df46894f964ab330" exitCode=0
Nov 03 20:03:38.958804 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:38.958622    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4" event={"ID":"811cc1b2-8239-4b7f-ab01-08eba26baed8","Type":"ContainerDied","Data":"665d2783255fca8e1c89527b8f8b3d98f099743bade2e132df46894f964ab330"}
Nov 03 20:03:38.973268 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:38.973226    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:03:39.468257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:39.465766    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:40.149206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.141036    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:40.212003 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.205218    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:40.314745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.314660    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-xks-ca\") pod \"811cc1b2-8239-4b7f-ab01-08eba26baed8\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") "
Nov 03 20:03:40.314745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.314708    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-openshift-ca\") pod \"811cc1b2-8239-4b7f-ab01-08eba26baed8\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") "
Nov 03 20:03:40.314949 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.314790    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811cc1b2-8239-4b7f-ab01-08eba26baed8-kube-api-access-r8j6t\" (UniqueName: \"kubernetes.io/projected/811cc1b2-8239-4b7f-ab01-08eba26baed8-kube-api-access-r8j6t\") pod \"811cc1b2-8239-4b7f-ab01-08eba26baed8\" (UID: \"811cc1b2-8239-4b7f-ab01-08eba26baed8\") "
Nov 03 20:03:40.315144 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.315119    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-openshift-ca" pod "811cc1b2-8239-4b7f-ab01-08eba26baed8" (UID: "811cc1b2-8239-4b7f-ab01-08eba26baed8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:40.315729 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.315709    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:40.316875 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.316852    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-xks-ca" pod "811cc1b2-8239-4b7f-ab01-08eba26baed8" (UID: "811cc1b2-8239-4b7f-ab01-08eba26baed8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:03:40.317016 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.316999    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811cc1b2-8239-4b7f-ab01-08eba26baed8-kube-api-access-r8j6t" pod "811cc1b2-8239-4b7f-ab01-08eba26baed8" (UID: "811cc1b2-8239-4b7f-ab01-08eba26baed8"). InnerVolumeSpecName "kube-api-access-r8j6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:40.417218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.417186    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811cc1b2-8239-4b7f-ab01-08eba26baed8-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:40.417218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.417216    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8j6t\" (UniqueName: \"kubernetes.io/projected/811cc1b2-8239-4b7f-ab01-08eba26baed8-kube-api-access-r8j6t\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:40.753197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.748778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:40.979712 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.979668    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4" event={"ID":"811cc1b2-8239-4b7f-ab01-08eba26baed8","Type":"ContainerDied","Data":"a996985de77ce8b37862541dc8d9185186b4f9a8c6225736597a0ba9053a0b2d"}
Nov 03 20:03:40.979892 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.979730    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a996985de77ce8b37862541dc8d9185186b4f9a8c6225736597a0ba9053a0b2d"
Nov 03 20:03:40.979892 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:40.979689    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"
Nov 03 20:03:41.441284 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:41.440964    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:42.328971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:42.328934    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:42.532670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:42.532634    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:43.278024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:43.277989    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:43.412584 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:43.412547    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:43.693481 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:43.693445    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:44.074165 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.073875    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"]
Nov 03 20:03:44.077649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.076042    4531 cpu_manager.go:414] "containerMap: removing container" podUID="811cc1b2-8239-4b7f-ab01-08eba26baed8" containerName="revoke-keys"
Nov 03 20:03:44.077649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.076071    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811cc1b2-8239-4b7f-ab01-08eba26baed8" containerName="revoke-keys"
Nov 03 20:03:44.077649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.076736    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="811cc1b2-8239-4b7f-ab01-08eba26baed8" containerName="revoke-keys"
Nov 03 20:03:44.086414 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.086269    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.096232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.095335    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"]
Nov 03 20:03:44.096747 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.096598    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-b30676-openshift-default-dockercfg-qk989\""
Nov 03 20:03:44.212897 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.212858    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:44.248192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.229997    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.248192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.230089    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-envoy\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.248192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.230141    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-credential-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.248192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.230295    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-certs\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.248192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.230334    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istiod-ca-cert\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.248192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.230503    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-token\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.248192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.230601    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-podinfo\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.248192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.230644    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-data\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.248192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.230670    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fkkv\" (UniqueName: \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-kube-api-access-9fkkv\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.340560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.340459    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-token\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.340888 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.340861    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-podinfo\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.341049 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.341031    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-data\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.341292 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.341261    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fkkv\" (UniqueName: \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-kube-api-access-9fkkv\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.341549 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.341519    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.341724 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.341706    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-envoy\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.341865 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.341851    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-credential-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.342100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.342084    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-certs\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.342251 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.342236    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istiod-ca-cert\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.359196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.343489    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istiod-ca-cert\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.361375 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.361342    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.361860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.361829    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-data\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.378195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.362562    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-credential-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.384659 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.363665    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-certs\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.429188 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.427941    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-envoy\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.430565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.430442    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-token\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.468736 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.459477    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fkkv\" (UniqueName: \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-kube-api-access-9fkkv\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.483216 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.480621    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:44.503552 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.503515    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-podinfo\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:44.616716 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.616626    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:44.720960 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:44.720924    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:45.012994 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:45.012960    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"]
Nov 03 20:03:45.013145 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:03:45.013003    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e747c83_6f8d_47ac_8023_d86ce46bcf94.slice/crio-778af0998d328be0b08a735be97bbfd66eb824bdcb3268eafb4d42ee885b1b2e WatchSource:0}: Error finding container 778af0998d328be0b08a735be97bbfd66eb824bdcb3268eafb4d42ee885b1b2e: Status 404 returned error can't find the container with id 778af0998d328be0b08a735be97bbfd66eb824bdcb3268eafb4d42ee885b1b2e
Nov 03 20:03:45.016202 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:45.016145    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:45.016333 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:45.016236    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:45.016333 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:45.016266    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:45.337185 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:45.337082    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:45.924283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:45.924138    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"]
Nov 03 20:03:45.991511 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:45.991478    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:46.080598 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:46.080555    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" event={"ID":"7e747c83-6f8d-47ac-8023-d86ce46bcf94","Type":"ContainerStarted","Data":"58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982"}
Nov 03 20:03:46.080970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:46.080941    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" event={"ID":"7e747c83-6f8d-47ac-8023-d86ce46bcf94","Type":"ContainerStarted","Data":"778af0998d328be0b08a735be97bbfd66eb824bdcb3268eafb4d42ee885b1b2e"}
Nov 03 20:03:46.109310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:46.109245    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" podStartSLOduration=2.109223804 podStartE2EDuration="2.109223804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:46.107932165 +0000 UTC m=+1939.378055463" watchObservedRunningTime="2026-11-03 20:03:46.109223804 +0000 UTC m=+1939.379347101"
Nov 03 20:03:46.159667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:46.159623    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:46.721601 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:46.721558    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:46.728305 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:46.728273    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:47.088684 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.088602    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:47.089768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.089745    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:03:47.401063 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.397153    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:47.908866 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.908819    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"]
Nov 03 20:03:47.928722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.928686    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"]
Nov 03 20:03:47.928936 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.928815    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:47.933727 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.933582    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3ff278\"/\"default-dockercfg-fq6bb\""
Nov 03 20:03:47.933727 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.933671    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3ff278\"/\"openshift-service-ca.crt\""
Nov 03 20:03:47.933727 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.933690    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3ff278\"/\"auth-test-model-3ff278-kserve-self-signed-certs\""
Nov 03 20:03:47.934033 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.933760    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3ff278\"/\"kube-root-ca.crt\""
Nov 03 20:03:47.955614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.955574    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tmp-dir\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:47.956106 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.955693    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-model-cache\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:47.956106 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.955924    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8wkz\" (UniqueName: \"kubernetes.io/projected/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-kube-api-access-l8wkz\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:47.956106 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.955954    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-dshm\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:47.956106 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.956000    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-home\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:47.956106 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.956089    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tls-certs\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:47.990523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:47.990478    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.058795 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.058744    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tmp-dir\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.059025 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.058848    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-model-cache\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.059025 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.059017    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8wkz\" (UniqueName: \"kubernetes.io/projected/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-kube-api-access-l8wkz\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.059345 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.059054    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-dshm\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.059345 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.059109    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-home\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.059345 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.059243    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tls-certs\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.059345 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.059302    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tmp-dir\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.059345 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.059326    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-model-cache\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.059699 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.059673    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-home\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.061552 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.061529    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-dshm\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.062046 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.062018    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tls-certs\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.066954 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.066928    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8wkz\" (UniqueName: \"kubernetes.io/projected/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-kube-api-access-l8wkz\") pod \"auth-test-model-3ff278-kserve-5899d7476f-7pcxs\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") " pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.259411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.255518    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:48.374195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.372782    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.666632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.666515    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.812461 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.812423    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.841870 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:03:48.841830    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda841ef7e_4a8d_469b_b13e_4d84980e3b1c.slice/crio-d530f0759a432aff2b74374fc3201e8bba62e9f78722a31c2555aaa9b4952f0b WatchSource:0}: Error finding container d530f0759a432aff2b74374fc3201e8bba62e9f78722a31c2555aaa9b4952f0b: Status 404 returned error can't find the container with id d530f0759a432aff2b74374fc3201e8bba62e9f78722a31c2555aaa9b4952f0b
Nov 03 20:03:48.842578 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:48.842544    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"]
Nov 03 20:03:49.108691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.108652    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs" event={"ID":"a841ef7e-4a8d-469b-b13e-4d84980e3b1c","Type":"ContainerStarted","Data":"2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31"}
Nov 03 20:03:49.108933 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.108703    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs" event={"ID":"a841ef7e-4a8d-469b-b13e-4d84980e3b1c","Type":"ContainerStarted","Data":"d530f0759a432aff2b74374fc3201e8bba62e9f78722a31c2555aaa9b4952f0b"}
Nov 03 20:03:49.131035 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.130960    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs" podStartSLOduration=2.130936135 podStartE2EDuration="2.130936135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:49.128975769 +0000 UTC m=+1942.399099065" watchObservedRunningTime="2026-11-03 20:03:49.130936135 +0000 UTC m=+1942.401059432"
Nov 03 20:03:49.179711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.171934    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:49.265198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.263209    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:49.412480 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.406013    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"]
Nov 03 20:03:49.473800 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.467418    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:49.473800 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.467311    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"]
Nov 03 20:03:49.475993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.475542    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:03:49.475993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.475572    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:03:49.506061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.506026    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:49.506282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.506075    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stpqh\" (UniqueName: \"kubernetes.io/projected/e8e4b9f4-67da-4926-9402-e7f82a5805ec-kube-api-access-stpqh\") pod \"maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:49.506282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.506150    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:49.528597 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.528557    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:49.608857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.608016    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:49.608857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.608084    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stpqh\" (UniqueName: \"kubernetes.io/projected/e8e4b9f4-67da-4926-9402-e7f82a5805ec-kube-api-access-stpqh\") pod \"maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:49.608857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.608133    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:49.623239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.611701    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:49.653198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.644465    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stpqh\" (UniqueName: \"kubernetes.io/projected/e8e4b9f4-67da-4926-9402-e7f82a5805ec-kube-api-access-stpqh\") pod \"maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:49.686260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.683580    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:49.796259 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:49.796150    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:50.102631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:50.102534    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"]
Nov 03 20:03:50.107362 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:03:50.107325    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e4b9f4_67da_4926_9402_e7f82a5805ec.slice/crio-d8b47bc8b0df4c9b09fccde4ff4f8ed2b7ac3911c6ccd667c6162909304efaed WatchSource:0}: Error finding container d8b47bc8b0df4c9b09fccde4ff4f8ed2b7ac3911c6ccd667c6162909304efaed: Status 404 returned error can't find the container with id d8b47bc8b0df4c9b09fccde4ff4f8ed2b7ac3911c6ccd667c6162909304efaed
Nov 03 20:03:50.122250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:50.122208    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d" event={"ID":"e8e4b9f4-67da-4926-9402-e7f82a5805ec","Type":"ContainerStarted","Data":"d8b47bc8b0df4c9b09fccde4ff4f8ed2b7ac3911c6ccd667c6162909304efaed"}
Nov 03 20:03:50.175898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:50.175862    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:50.315200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:50.310126    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:50.878073 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:03:50.878006    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:03:50.912488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:50.912451    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:51.132537 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:51.132498    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d" event={"ID":"e8e4b9f4-67da-4926-9402-e7f82a5805ec","Type":"ContainerStarted","Data":"2b427e29beb241d47623a53361a9f4ada9f2e7049a30ccd3c9c8073ce03c4df9"}
Nov 03 20:03:51.158940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:51.158857    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d" podStartSLOduration=2.158837047 podStartE2EDuration="2.158837047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:51.146878277 +0000 UTC m=+1944.417001578" watchObservedRunningTime="2026-11-03 20:03:51.158837047 +0000 UTC m=+1944.428960344"
Nov 03 20:03:51.759112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:51.758999    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:52.227194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:52.226596    4531 generic.go:356] "Generic (PLEG): container finished" podID="e8e4b9f4-67da-4926-9402-e7f82a5805ec" containerID="2b427e29beb241d47623a53361a9f4ada9f2e7049a30ccd3c9c8073ce03c4df9" exitCode=0
Nov 03 20:03:52.227194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:52.226861    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d" event={"ID":"e8e4b9f4-67da-4926-9402-e7f82a5805ec","Type":"ContainerDied","Data":"2b427e29beb241d47623a53361a9f4ada9f2e7049a30ccd3c9c8073ce03c4df9"}
Nov 03 20:03:53.433676 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:53.433647    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:53.510464 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:53.510384    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-xks-ca\") pod \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") "
Nov 03 20:03:53.510626 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:53.510496    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e4b9f4-67da-4926-9402-e7f82a5805ec-kube-api-access-stpqh\" (UniqueName: \"kubernetes.io/projected/e8e4b9f4-67da-4926-9402-e7f82a5805ec-kube-api-access-stpqh\") pod \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") "
Nov 03 20:03:53.510710 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:53.510694    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-openshift-ca\") pod \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\" (UID: \"e8e4b9f4-67da-4926-9402-e7f82a5805ec\") "
Nov 03 20:03:53.511060 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:53.511034    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-openshift-ca" pod "e8e4b9f4-67da-4926-9402-e7f82a5805ec" (UID: "e8e4b9f4-67da-4926-9402-e7f82a5805ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:53.511498 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:53.511477    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:53.512652 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:53.512623    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-xks-ca" pod "e8e4b9f4-67da-4926-9402-e7f82a5805ec" (UID: "e8e4b9f4-67da-4926-9402-e7f82a5805ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:03:53.512774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:53.512701    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e4b9f4-67da-4926-9402-e7f82a5805ec-kube-api-access-stpqh" pod "e8e4b9f4-67da-4926-9402-e7f82a5805ec" (UID: "e8e4b9f4-67da-4926-9402-e7f82a5805ec"). InnerVolumeSpecName "kube-api-access-stpqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:53.613283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:53.613245    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e4b9f4-67da-4926-9402-e7f82a5805ec-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:53.613283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:53.613281    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stpqh\" (UniqueName: \"kubernetes.io/projected/e8e4b9f4-67da-4926-9402-e7f82a5805ec-kube-api-access-stpqh\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:53.799255 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:53.797083    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:54.247798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:54.247768    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"
Nov 03 20:03:54.247997 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:54.247799    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d" event={"ID":"e8e4b9f4-67da-4926-9402-e7f82a5805ec","Type":"ContainerDied","Data":"d8b47bc8b0df4c9b09fccde4ff4f8ed2b7ac3911c6ccd667c6162909304efaed"}
Nov 03 20:03:54.247997 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:54.247846    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8b47bc8b0df4c9b09fccde4ff4f8ed2b7ac3911c6ccd667c6162909304efaed"
Nov 03 20:03:55.160200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:55.155779    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:56.435484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.430422    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:56.454399 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.449606    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"]
Nov 03 20:03:56.463205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.462558    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"]
Nov 03 20:03:56.583293 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.583257    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:56.696686 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.696595    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"]
Nov 03 20:03:56.697993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.697963    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e8e4b9f4-67da-4926-9402-e7f82a5805ec" containerName="revoke-keys"
Nov 03 20:03:56.697993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.697991    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e4b9f4-67da-4926-9402-e7f82a5805ec" containerName="revoke-keys"
Nov 03 20:03:56.705312 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.698245    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e4b9f4-67da-4926-9402-e7f82a5805ec" containerName="revoke-keys"
Nov 03 20:03:56.709609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.709581    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"]
Nov 03 20:03:56.709728 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.709697    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.712430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.712402    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-32a584\"/\"default-dockercfg-2k54c\""
Nov 03 20:03:56.712641 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.712586    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-32a584\"/\"auth-test-model-32a584-kserve-self-signed-certs\""
Nov 03 20:03:56.712706 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.712648    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-32a584\"/\"kube-root-ca.crt\""
Nov 03 20:03:56.712749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.712705    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-32a584\"/\"openshift-service-ca.crt\""
Nov 03 20:03:56.773635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.773583    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:56.791938 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.791893    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-model-cache\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.792192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.792038    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-tmp-dir\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.792352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.792219    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-dshm\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.792352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.792306    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-home\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.792458 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.792359    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpphq\" (UniqueName: \"kubernetes.io/projected/79a95827-f3b5-4c1d-8c1f-419924011fe6-kube-api-access-vpphq\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.792517 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.792457    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a95827-f3b5-4c1d-8c1f-419924011fe6-tls-certs\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.920197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.917539    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-dshm\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.920197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.917666    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-home\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.920197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.917707    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpphq\" (UniqueName: \"kubernetes.io/projected/79a95827-f3b5-4c1d-8c1f-419924011fe6-kube-api-access-vpphq\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.920197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.917768    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a95827-f3b5-4c1d-8c1f-419924011fe6-tls-certs\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.920197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.917980    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-model-cache\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.920197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.918074    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-tmp-dir\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.922802 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.921449    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-home\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.922802 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.921711    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-model-cache\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.922802 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.922386    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-tmp-dir\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.961730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.937102    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-dshm\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.961730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.942112    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpphq\" (UniqueName: \"kubernetes.io/projected/79a95827-f3b5-4c1d-8c1f-419924011fe6-kube-api-access-vpphq\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:56.981886 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:56.981843    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a95827-f3b5-4c1d-8c1f-419924011fe6-tls-certs\") pod \"auth-test-model-32a584-kserve-bc8cc974f-8cmlc\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") " pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:57.035599 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:57.035555    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:57.481377 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:57.481341    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"]
Nov 03 20:03:57.759839 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:57.759731    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:58.256080 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:58.256046    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:58.268689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:58.268664    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:58.310512 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:58.310472    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc" event={"ID":"79a95827-f3b5-4c1d-8c1f-419924011fe6","Type":"ContainerStarted","Data":"4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084"}
Nov 03 20:03:58.310706 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:58.310523    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc" event={"ID":"79a95827-f3b5-4c1d-8c1f-419924011fe6","Type":"ContainerStarted","Data":"21fabbb413080afee5e10869db6cac048bd4b3bfab5b84f566639f0e48bf183e"}
Nov 03 20:03:58.325318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:58.325281    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:03:58.326645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:58.326603    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc" podStartSLOduration=2.326589738 podStartE2EDuration="2.326589738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:58.324121544 +0000 UTC m=+1951.594244855" watchObservedRunningTime="2026-11-03 20:03:58.326589738 +0000 UTC m=+1951.596713033"
Nov 03 20:03:59.036939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:59.036904    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:03:59.137940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:59.137885    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:59.641781 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:59.641743    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:59.765131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:03:59.765092    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:00.620555 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:00.620509    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:00.954452 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:00.954391    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl" podUID="1d818361-422a-4eb2-a643-08dc21861021" containerName="main" containerID="cri-o://42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0" gracePeriod=45
Nov 03 20:04:01.278324 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.278295    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 20:04:01.341384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.341342    4531 generic.go:356] "Generic (PLEG): container finished" podID="1d818361-422a-4eb2-a643-08dc21861021" containerID="42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0" exitCode=0
Nov 03 20:04:01.341596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.341410    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"
Nov 03 20:04:01.341596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.341420    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl" event={"ID":"1d818361-422a-4eb2-a643-08dc21861021","Type":"ContainerDied","Data":"42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0"}
Nov 03 20:04:01.341596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.341469    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl" event={"ID":"1d818361-422a-4eb2-a643-08dc21861021","Type":"ContainerDied","Data":"b989838003a78a509dd74af66ea8bd019242c0034a9bf942e1d35b6e84987681"}
Nov 03 20:04:01.341596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.341491    4531 scope.go:122] "RemoveContainer" containerID="42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0"
Nov 03 20:04:01.354301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.354282    4531 scope.go:122] "RemoveContainer" containerID="42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0"
Nov 03 20:04:01.354620 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:01.354589    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0\": container with ID starting with 42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0 not found: ID does not exist" containerID="42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0"
Nov 03 20:04:01.354705 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.354632    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0"} err="failed to get container status \"42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0\": rpc error: code = NotFound desc = could not find container \"42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0\": container with ID starting with 42be0680c91f4e8b906488c21cbc276869176cc69bb97b4f8db07deb08353ad0 not found: ID does not exist"
Nov 03 20:04:01.382583 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.382552    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-dshm\") pod \"1d818361-422a-4eb2-a643-08dc21861021\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") "
Nov 03 20:04:01.382725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.382694    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-model-cache\") pod \"1d818361-422a-4eb2-a643-08dc21861021\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") "
Nov 03 20:04:01.382770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.382732    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-tmp-dir\") pod \"1d818361-422a-4eb2-a643-08dc21861021\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") "
Nov 03 20:04:01.382817 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.382767    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-home\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-home\") pod \"1d818361-422a-4eb2-a643-08dc21861021\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") "
Nov 03 20:04:01.382945 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.382925    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d818361-422a-4eb2-a643-08dc21861021-kube-api-access-zp84s\" (UniqueName: \"kubernetes.io/projected/1d818361-422a-4eb2-a643-08dc21861021-kube-api-access-zp84s\") pod \"1d818361-422a-4eb2-a643-08dc21861021\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") "
Nov 03 20:04:01.383021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.382965    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d818361-422a-4eb2-a643-08dc21861021-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d818361-422a-4eb2-a643-08dc21861021-tls-certs\") pod \"1d818361-422a-4eb2-a643-08dc21861021\" (UID: \"1d818361-422a-4eb2-a643-08dc21861021\") "
Nov 03 20:04:01.383021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.382966    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-model-cache" pod "1d818361-422a-4eb2-a643-08dc21861021" (UID: "1d818361-422a-4eb2-a643-08dc21861021"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:01.383139 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.383023    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-tmp-dir" pod "1d818361-422a-4eb2-a643-08dc21861021" (UID: "1d818361-422a-4eb2-a643-08dc21861021"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:01.383139 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.383074    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-home" pod "1d818361-422a-4eb2-a643-08dc21861021" (UID: "1d818361-422a-4eb2-a643-08dc21861021"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:01.383872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.383847    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:01.383960 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.383874    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:01.383960 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.383890    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:01.384958 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.384931    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-dshm" pod "1d818361-422a-4eb2-a643-08dc21861021" (UID: "1d818361-422a-4eb2-a643-08dc21861021"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:01.385049 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.385002    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d818361-422a-4eb2-a643-08dc21861021-kube-api-access-zp84s" pod "1d818361-422a-4eb2-a643-08dc21861021" (UID: "1d818361-422a-4eb2-a643-08dc21861021"). InnerVolumeSpecName "kube-api-access-zp84s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:01.385049 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.385006    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d818361-422a-4eb2-a643-08dc21861021-tls-certs" pod "1d818361-422a-4eb2-a643-08dc21861021" (UID: "1d818361-422a-4eb2-a643-08dc21861021"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:01.457226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.450706    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4" podUID="762a9a66-e013-496b-b744-dc20fe6d4b13" containerName="payload-processing" containerID="cri-o://43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a" gracePeriod=25
Nov 03 20:04:01.457226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.453831    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:01.478511 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.463889    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb" podUID="e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3" containerName="payload-pre-processing" containerID="cri-o://ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b" gracePeriod=25
Nov 03 20:04:01.496358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.494559    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp84s\" (UniqueName: \"kubernetes.io/projected/1d818361-422a-4eb2-a643-08dc21861021-kube-api-access-zp84s\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:01.496358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.494601    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d818361-422a-4eb2-a643-08dc21861021-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:01.496358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.494617    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d818361-422a-4eb2-a643-08dc21861021-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:01.666793 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.666759    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"]
Nov 03 20:04:01.671669 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:01.671649    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl"]
Nov 03 20:04:02.160200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:02.159783    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:02.229154 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:02.229116    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d12868/test-model-d12868-kserve-5db7944f69-fsrfl" podUID="1d818361-422a-4eb2-a643-08dc21861021" containerName="main" probeResult="failure" output="Get \"https://10.133.0.212:8000/ready\": context deadline exceeded"
Nov 03 20:04:02.861392 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:02.861355    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d818361-422a-4eb2-a643-08dc21861021" path="/var/lib/kubelet/pods/1d818361-422a-4eb2-a643-08dc21861021/volumes"
Nov 03 20:04:02.961202 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:02.958231    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:03.019841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:03.019798    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4" podUID="762a9a66-e013-496b-b744-dc20fe6d4b13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:04:03.025488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:03.021399    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb" podUID="e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:04:03.799715 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:03.799673    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:04.555126 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:04.555083    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:05.055100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.055061    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"]
Nov 03 20:04:05.055378 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.055349    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d" podUID="2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a" containerName="maas-api" containerID="cri-o://896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3" gracePeriod=30
Nov 03 20:04:05.140911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.140872    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:05.345288 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.336495    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:05.636855 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.636772    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:05.743199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.740451    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 20:04:05.781667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.778654    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:05.826367 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.826328    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-metrics-certs\") pod \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") "
Nov 03 20:04:05.826587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.826476    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-kube-api-access-gcc8j\" (UniqueName: \"kubernetes.io/projected/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-kube-api-access-gcc8j\") pod \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") "
Nov 03 20:04:05.826672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.826587    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-maas-api-tls\") pod \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") "
Nov 03 20:04:05.826887 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.826865    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-openshift-service-ca\") pod \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\" (UID: \"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a\") "
Nov 03 20:04:05.827317 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.827288    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-openshift-service-ca" pod "2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a" (UID: "2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:05.827635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.827607    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:05.829464 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.829435    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-kube-api-access-gcc8j" pod "2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a" (UID: "2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a"). InnerVolumeSpecName "kube-api-access-gcc8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:05.829628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.829585    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-metrics-certs" pod "2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a" (UID: "2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:05.829753 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.829674    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-maas-api-tls" pod "2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a" (UID: "2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:05.929518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.929420    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:05.929518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.929456    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcc8j\" (UniqueName: \"kubernetes.io/projected/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-kube-api-access-gcc8j\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:05.929518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:05.929471    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:06.095078 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.095047    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:06.458816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.458776    4531 generic.go:356] "Generic (PLEG): container finished" podID="2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a" containerID="896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3" exitCode=0
Nov 03 20:04:06.459040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.458830    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"
Nov 03 20:04:06.459040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.458835    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d" event={"ID":"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a","Type":"ContainerDied","Data":"896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3"}
Nov 03 20:04:06.459040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.458901    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d" event={"ID":"2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a","Type":"ContainerDied","Data":"dff5c0a62c522f935e8d4464b06d6e9a73e2e33f68a1d35b72b9482a20d887fd"}
Nov 03 20:04:06.459040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.458925    4531 scope.go:122] "RemoveContainer" containerID="896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3"
Nov 03 20:04:06.472450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.472430    4531 scope.go:122] "RemoveContainer" containerID="896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3"
Nov 03 20:04:06.472759 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:06.472738    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3\": container with ID starting with 896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3 not found: ID does not exist" containerID="896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3"
Nov 03 20:04:06.472814 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.472767    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3"} err="failed to get container status \"896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3\": rpc error: code = NotFound desc = could not find container \"896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3\": container with ID starting with 896da598e3cf7773693fe9ddf68738c7661011ae02cbd4d1f3b59fa1f943d2f3 not found: ID does not exist"
Nov 03 20:04:06.483716 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.483684    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"]
Nov 03 20:04:06.488254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.488229    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7a62d4-5d844b5588-2vl9d"]
Nov 03 20:04:06.649603 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.649560    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:06.869709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:06.869620    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a" path="/var/lib/kubelet/pods/2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a/volumes"
Nov 03 20:04:07.037038 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.037007    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:04:07.052268 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.052240    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:04:07.292381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.292346    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"]
Nov 03 20:04:07.293212 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.293189    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a" containerName="maas-api"
Nov 03 20:04:07.293212 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.293210    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a" containerName="maas-api"
Nov 03 20:04:07.293423 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.293230    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1d818361-422a-4eb2-a643-08dc21861021" containerName="main"
Nov 03 20:04:07.293423 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.293236    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d818361-422a-4eb2-a643-08dc21861021" containerName="main"
Nov 03 20:04:07.293423 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.293406    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2692cd61-b0a3-4a74-9f4e-3f4c1ca4e35a" containerName="maas-api"
Nov 03 20:04:07.293423 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.293416    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d818361-422a-4eb2-a643-08dc21861021" containerName="main"
Nov 03 20:04:07.299185 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.299143    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.304511 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.304484    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-dce558\"/\"kube-root-ca.crt\""
Nov 03 20:04:07.304729 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.304683    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-dce558\"/\"openshift-service-ca.crt\""
Nov 03 20:04:07.304845 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.304690    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-dce558\"/\"auth-test-model-dce558-kserve-self-signed-certs\""
Nov 03 20:04:07.304845 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.304736    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-dce558\"/\"default-dockercfg-hl6xt\""
Nov 03 20:04:07.306743 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.306720    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"]
Nov 03 20:04:07.386768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.386735    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:07.525197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.513790    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-home\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.525197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.513889    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tmp-dir\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.525197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.514052    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tls-certs\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.525197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.514142    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-dshm\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.525197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.514233    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-model-cache\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.525197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.514265    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6tk\" (UniqueName: \"kubernetes.io/projected/8545f1c7-81a0-42a1-871e-47ca9a231c2d-kube-api-access-jw6tk\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.536672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.535650    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:07.621241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.617508    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tls-certs\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.621241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.617627    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-dshm\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.621241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.617672    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-model-cache\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.621241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.617702    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6tk\" (UniqueName: \"kubernetes.io/projected/8545f1c7-81a0-42a1-871e-47ca9a231c2d-kube-api-access-jw6tk\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.621241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.617861    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-home\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.621241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.617957    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tmp-dir\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.621241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.618608    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tmp-dir\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.653001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.647583    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-model-cache\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.653001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.648489    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-home\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.653001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.652331    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tls-certs\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.659195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.654131    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-dshm\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.659195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.657642    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6tk\" (UniqueName: \"kubernetes.io/projected/8545f1c7-81a0-42a1-871e-47ca9a231c2d-kube-api-access-jw6tk\") pod \"auth-test-model-dce558-kserve-74695d5dd5-xnv6c\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") " pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.670596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.661683    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:04:07.821853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.821819    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"]
Nov 03 20:04:07.826942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.826919    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:07.834974 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.834671    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-dce558-openshift-default-dockercfg-rshjr\""
Nov 03 20:04:07.836129 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.836103    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"]
Nov 03 20:04:07.918432 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.918397    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:07.933220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.930191    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-certs\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:07.933220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.932759    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-data\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:07.933710 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.933594    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psxf7\" (UniqueName: \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-kube-api-access-psxf7\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:07.933879 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.933854    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istiod-ca-cert\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:07.934204 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.934165    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:07.934301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.934284    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-token\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:07.934421 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.934402    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-credential-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:07.934493 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.934443    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-podinfo\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:07.934549 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:07.934517    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-envoy\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.040882    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-certs\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.040990    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-data\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.041963    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-certs\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.041047    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxf7\" (UniqueName: \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-kube-api-access-psxf7\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.042347    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istiod-ca-cert\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.042488    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.042633    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-token\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.042742    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-credential-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.042775    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-podinfo\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.042840    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-envoy\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.046296    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-data\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.047294    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-credential-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.047906 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.047784    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istiod-ca-cert\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.049359 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.048998    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.056785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.056737    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-envoy\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.057710 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.057050    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-token\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.059515 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.059103    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxf7\" (UniqueName: \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-kube-api-access-psxf7\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.060220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.059970    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-podinfo\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.149294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.149262    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:08.236759 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.236727    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"]
Nov 03 20:04:08.245787 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:04:08.245753    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8545f1c7_81a0_42a1_871e_47ca9a231c2d.slice/crio-86e0d41fcd52283d6b860691ca47b758b385b6a4c434122517669c3e105fe42e WatchSource:0}: Error finding container 86e0d41fcd52283d6b860691ca47b758b385b6a4c434122517669c3e105fe42e: Status 404 returned error can't find the container with id 86e0d41fcd52283d6b860691ca47b758b385b6a4c434122517669c3e105fe42e
Nov 03 20:04:08.336662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.336499    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:08.440316 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:04:08.440284    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bcb497_8b2e_4fcd_9e14_110914fe6cde.slice/crio-b815691762cfcd7f70b3a5b4ec2495944b77c10658fbc251173bf9fd4f9773a7 WatchSource:0}: Error finding container b815691762cfcd7f70b3a5b4ec2495944b77c10658fbc251173bf9fd4f9773a7: Status 404 returned error can't find the container with id b815691762cfcd7f70b3a5b4ec2495944b77c10658fbc251173bf9fd4f9773a7
Nov 03 20:04:08.442225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.442201    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"]
Nov 03 20:04:08.443655 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.443621    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:08.443733 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.443690    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:08.443733 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.443718    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:08.549194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.547099    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" event={"ID":"c9bcb497-8b2e-4fcd-9e14-110914fe6cde","Type":"ContainerStarted","Data":"124c9effe4b79c2d4ca27b95b757a7c046ddfbd4b407cd5a321a8de1a2e67a77"}
Nov 03 20:04:08.549194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.547157    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" event={"ID":"c9bcb497-8b2e-4fcd-9e14-110914fe6cde","Type":"ContainerStarted","Data":"b815691762cfcd7f70b3a5b4ec2495944b77c10658fbc251173bf9fd4f9773a7"}
Nov 03 20:04:08.549194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.549054    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c" event={"ID":"8545f1c7-81a0-42a1-871e-47ca9a231c2d","Type":"ContainerStarted","Data":"43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f"}
Nov 03 20:04:08.549194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.549092    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c" event={"ID":"8545f1c7-81a0-42a1-871e-47ca9a231c2d","Type":"ContainerStarted","Data":"86e0d41fcd52283d6b860691ca47b758b385b6a4c434122517669c3e105fe42e"}
Nov 03 20:04:08.566260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.566168    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" podStartSLOduration=1.566152504 podStartE2EDuration="1.566152504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:08.563240853 +0000 UTC m=+1961.833364151" watchObservedRunningTime="2026-11-03 20:04:08.566152504 +0000 UTC m=+1961.836275799"
Nov 03 20:04:08.578592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.578543    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c" podStartSLOduration=1.578529901 podStartE2EDuration="1.578529901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:08.576438838 +0000 UTC m=+1961.846562132" watchObservedRunningTime="2026-11-03 20:04:08.578529901 +0000 UTC m=+1961.848653196"
Nov 03 20:04:08.919707 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:08.919670    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:09.124318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:09.124280    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:09.157711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:09.151526    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:09.172255 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:09.167319    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 20:04:09.172255 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:09.167385    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 20:04:10.160209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.159665    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 20:04:10.160209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.159740    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 20:04:10.299200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.295981    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:10.869982 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.869941    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:10.928415 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.928377    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"]
Nov 03 20:04:10.933781 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.933753    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:10.984931 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.984219    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:10.988670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.987434    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:10.988670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.987504    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:10.988670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.987541    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bnkg\" (UniqueName: \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-kube-api-access-7bnkg\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:10.988670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.987628    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:10.988670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.987665    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:10.988670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.987800    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:10.988670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.987839    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:10.988670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.987888    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:10.997965 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:10.997931    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"]
Nov 03 20:04:11.016368 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.016306    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:11.090074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090034    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090081    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090122    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090149    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bnkg\" (UniqueName: \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-kube-api-access-7bnkg\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090236    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090273    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090375    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090406    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090450    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090524    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090656    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090731    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.090853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090826    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.091019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.090964    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.092889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.092868    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.093140 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.093120    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.098570 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.098547    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bnkg\" (UniqueName: \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-kube-api-access-7bnkg\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.098679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.098588    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.161200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.160622    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 20:04:11.161200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.160705    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 20:04:11.301729 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.301692    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:11.635208 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.634281    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:11.683998 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.683959    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"]
Nov 03 20:04:11.695065 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:04:11.695001    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595c6bfd_72b7_4501_95ce_5c7e3eda018c.slice/crio-ecdd42645376188cb8e5be67ccdcaa20d463f9ebabbf6f39b261589588da4c01 WatchSource:0}: Error finding container ecdd42645376188cb8e5be67ccdcaa20d463f9ebabbf6f39b261589588da4c01: Status 404 returned error can't find the container with id ecdd42645376188cb8e5be67ccdcaa20d463f9ebabbf6f39b261589588da4c01
Nov 03 20:04:11.698912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.698879    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:11.699255 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.699231    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:11.699390 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:11.699377    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:12.157198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:12.152285    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 20:04:12.157198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:12.152352    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 20:04:12.306407 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:12.306375    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:12.469254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:12.467126    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:12.597197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:12.596658    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"]
Nov 03 20:04:12.597412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:12.597280    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" podUID="73124dfd-65ad-4cc7-b44d-8520be810d29" containerName="istio-proxy" containerID="cri-o://73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083" gracePeriod=30
Nov 03 20:04:12.652265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:12.645074    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:13.057587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.057534    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb" podUID="e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:04:13.065260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.064941    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4" podUID="762a9a66-e013-496b-b744-dc20fe6d4b13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:04:13.117264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.085086    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" event={"ID":"595c6bfd-72b7-4501-95ce-5c7e3eda018c","Type":"ContainerStarted","Data":"1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf"}
Nov 03 20:04:13.117264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.085153    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" event={"ID":"595c6bfd-72b7-4501-95ce-5c7e3eda018c","Type":"ContainerStarted","Data":"ecdd42645376188cb8e5be67ccdcaa20d463f9ebabbf6f39b261589588da4c01"}
Nov 03 20:04:13.159626 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.159583    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.159935 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.159899    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 20:04:13.201640 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.201574    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" podStartSLOduration=3.201556292 podStartE2EDuration="3.201556292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:13.154094183 +0000 UTC m=+1966.424217500" watchObservedRunningTime="2026-11-03 20:04:13.201556292 +0000 UTC m=+1966.471679589"
Nov 03 20:04:13.302258 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.302222    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:13.304462 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.304435    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.304655 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.304634    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 20:04:13.722975 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.722527    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:13.885271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:13.879747    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:14.161284 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:14.161187    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.161284 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:14.161254    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 20:04:14.282199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:14.281319    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:14.306379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:14.305732    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.306379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:14.305814    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 20:04:15.170200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:15.158291    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 20:04:15.170200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:15.158360    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 20:04:15.350225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:15.336037    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 20:04:15.350225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:15.336099    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 20:04:15.591509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:15.588528    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:16.164842 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:16.164803    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:16.165303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:16.165274    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:16.166034 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:16.166015    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:04:16.302330 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:16.302299    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 20:04:16.302706 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:16.302351    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 20:04:16.969197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:16.968620    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:17.099123 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:17.099093    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:17.303752 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:17.303664    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 20:04:17.304397 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:17.304357    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 20:04:17.827376 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:17.824397    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:17.978059 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:17.977968    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:18.030000 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.029973    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:18.091020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.090996    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:04:18.165768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.165733    4531 generic.go:356] "Generic (PLEG): container finished" podID="73124dfd-65ad-4cc7-b44d-8520be810d29" containerID="73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083" exitCode=0
Nov 03 20:04:18.167497 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.167475    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"
Nov 03 20:04:18.168099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.168076    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" event={"ID":"73124dfd-65ad-4cc7-b44d-8520be810d29","Type":"ContainerDied","Data":"73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083"}
Nov 03 20:04:18.168287 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.168267    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh" event={"ID":"73124dfd-65ad-4cc7-b44d-8520be810d29","Type":"ContainerDied","Data":"bcb5350b2a252c3a33d5eb54ec1e61fc6f13ff959835a0910681f0658f3aa36e"}
Nov 03 20:04:18.168411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.168397    4531 scope.go:122] "RemoveContainer" containerID="73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083"
Nov 03 20:04:18.188255 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.188224    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:04:18.188903 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.188855    4531 scope.go:122] "RemoveContainer" containerID="73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083"
Nov 03 20:04:18.189773 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:18.189725    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083\": container with ID starting with 73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083 not found: ID does not exist" containerID="73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083"
Nov 03 20:04:18.189773 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.189761    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083"} err="failed to get container status \"73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083\": rpc error: code = NotFound desc = could not find container \"73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083\": container with ID starting with 73d1f8fb2d6248c5972b31abf9cddeb74a28c36e4ce3194a84d7d52ed3853083 not found: ID does not exist"
Nov 03 20:04:18.199506 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.199479    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-certs\") pod \"73124dfd-65ad-4cc7-b44d-8520be810d29\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") "
Nov 03 20:04:18.199609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.199516    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-podinfo\") pod \"73124dfd-65ad-4cc7-b44d-8520be810d29\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") "
Nov 03 20:04:18.199609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.199601    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-envoy\") pod \"73124dfd-65ad-4cc7-b44d-8520be810d29\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") "
Nov 03 20:04:18.199732 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.199644    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-data\") pod \"73124dfd-65ad-4cc7-b44d-8520be810d29\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") "
Nov 03 20:04:18.199732 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.199676    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-kube-api-access-nlhns\" (UniqueName: \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-kube-api-access-nlhns\") pod \"73124dfd-65ad-4cc7-b44d-8520be810d29\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") "
Nov 03 20:04:18.199732 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.199710    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-credential-socket\") pod \"73124dfd-65ad-4cc7-b44d-8520be810d29\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") "
Nov 03 20:04:18.199895 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.199777    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-socket\") pod \"73124dfd-65ad-4cc7-b44d-8520be810d29\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") "
Nov 03 20:04:18.199895 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.199840    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-token\" (UniqueName: \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-token\") pod \"73124dfd-65ad-4cc7-b44d-8520be810d29\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") "
Nov 03 20:04:18.200012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.199995    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73124dfd-65ad-4cc7-b44d-8520be810d29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73124dfd-65ad-4cc7-b44d-8520be810d29-istiod-ca-cert\") pod \"73124dfd-65ad-4cc7-b44d-8520be810d29\" (UID: \"73124dfd-65ad-4cc7-b44d-8520be810d29\") "
Nov 03 20:04:18.200012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.199993    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-data" pod "73124dfd-65ad-4cc7-b44d-8520be810d29" (UID: "73124dfd-65ad-4cc7-b44d-8520be810d29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:18.200395 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.200371    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73124dfd-65ad-4cc7-b44d-8520be810d29-istiod-ca-cert" pod "73124dfd-65ad-4cc7-b44d-8520be810d29" (UID: "73124dfd-65ad-4cc7-b44d-8520be810d29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:18.200793 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.200772    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-socket" pod "73124dfd-65ad-4cc7-b44d-8520be810d29" (UID: "73124dfd-65ad-4cc7-b44d-8520be810d29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:18.201155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.200799    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-credential-socket" pod "73124dfd-65ad-4cc7-b44d-8520be810d29" (UID: "73124dfd-65ad-4cc7-b44d-8520be810d29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:18.201414 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.201390    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-certs" pod "73124dfd-65ad-4cc7-b44d-8520be810d29" (UID: "73124dfd-65ad-4cc7-b44d-8520be810d29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:18.202152 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.202130    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:18.202412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.202395    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73124dfd-65ad-4cc7-b44d-8520be810d29-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:18.203163 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.203138    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:18.203163 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.202894    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-envoy" pod "73124dfd-65ad-4cc7-b44d-8520be810d29" (UID: "73124dfd-65ad-4cc7-b44d-8520be810d29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:18.203360 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.203308    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-podinfo" pod "73124dfd-65ad-4cc7-b44d-8520be810d29" (UID: "73124dfd-65ad-4cc7-b44d-8520be810d29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:04:18.206369 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.206347    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-token" pod "73124dfd-65ad-4cc7-b44d-8520be810d29" (UID: "73124dfd-65ad-4cc7-b44d-8520be810d29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:18.206518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.206494    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-kube-api-access-nlhns" pod "73124dfd-65ad-4cc7-b44d-8520be810d29" (UID: "73124dfd-65ad-4cc7-b44d-8520be810d29"). InnerVolumeSpecName "kube-api-access-nlhns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:18.303021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.302937    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 20:04:18.303021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.302999    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 20:04:18.304381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.304351    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:18.304381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.304382    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:18.304760 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.304396    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:18.304760 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.304410    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlhns\" (UniqueName: \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-kube-api-access-nlhns\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:18.304760 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.304424    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73124dfd-65ad-4cc7-b44d-8520be810d29-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:18.304760 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.304437    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73124dfd-65ad-4cc7-b44d-8520be810d29-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:18.494666 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.494624    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"]
Nov 03 20:04:18.499618 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.499587    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-jrghh"]
Nov 03 20:04:18.639201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.633942    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:18.873051 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:18.873015    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73124dfd-65ad-4cc7-b44d-8520be810d29" path="/var/lib/kubelet/pods/73124dfd-65ad-4cc7-b44d-8520be810d29/volumes"
Nov 03 20:04:19.302275 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:19.302229    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 20:04:19.302501 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:19.302322    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 20:04:19.453765 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:19.453723    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:20.170667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.170616    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:20.307190 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.307144    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:20.307472 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.307449    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:20.308215 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.308160    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:20.574887 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:20.574796    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:04:20.593080 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.593040    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"]
Nov 03 20:04:20.593930 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.593908    4531 cpu_manager.go:414] "containerMap: removing container" podUID="73124dfd-65ad-4cc7-b44d-8520be810d29" containerName="istio-proxy"
Nov 03 20:04:20.594061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.593931    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73124dfd-65ad-4cc7-b44d-8520be810d29" containerName="istio-proxy"
Nov 03 20:04:20.594229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.594213    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="73124dfd-65ad-4cc7-b44d-8520be810d29" containerName="istio-proxy"
Nov 03 20:04:20.599836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.599811    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:20.603105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.603059    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:04:20.603278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.603155    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:04:20.609475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.609445    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"]
Nov 03 20:04:20.651731 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.651688    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b44a7a-gthpm\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:20.652082 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.652049    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8pb\" (UniqueName: \"kubernetes.io/projected/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-kube-api-access-9x8pb\") pod \"maas-api-revoke-sub-67b44a7a-gthpm\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:20.652256 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.652118    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b44a7a-gthpm\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:20.753881 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.753845    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b44a7a-gthpm\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:20.754161 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.754145    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b44a7a-gthpm\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:20.754264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.754239    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8pb\" (UniqueName: \"kubernetes.io/projected/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-kube-api-access-9x8pb\") pod \"maas-api-revoke-sub-67b44a7a-gthpm\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:20.754816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.754787    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67b44a7a-gthpm\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:20.756311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.756290    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67b44a7a-gthpm\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:20.762068 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.762046    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8pb\" (UniqueName: \"kubernetes.io/projected/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-kube-api-access-9x8pb\") pod \"maas-api-revoke-sub-67b44a7a-gthpm\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:20.826819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.826724    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:20.920719 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:20.920687    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:21.234916 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:04:21.234885    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00be0ef0_9ee6_4f14_a738_523b34ebb8bd.slice/crio-43d3e35df0840bd26f1a390232a2ac218220c05a6cbe513d7ec2b4c696e2268e WatchSource:0}: Error finding container 43d3e35df0840bd26f1a390232a2ac218220c05a6cbe513d7ec2b4c696e2268e: Status 404 returned error can't find the container with id 43d3e35df0840bd26f1a390232a2ac218220c05a6cbe513d7ec2b4c696e2268e
Nov 03 20:04:21.235711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:21.235685    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"]
Nov 03 20:04:21.739668 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:21.726873    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:21.873796 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:21.861288    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"]
Nov 03 20:04:21.915976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:21.914914    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:22.152302 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.151037    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 20:04:22.152302 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.151432    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"]
Nov 03 20:04:22.155889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.153132    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3" containerName="payload-pre-processing"
Nov 03 20:04:22.155889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.153155    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3" containerName="payload-pre-processing"
Nov 03 20:04:22.155889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.153451    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3" containerName="payload-pre-processing"
Nov 03 20:04:22.160780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.160758    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.166272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.163912    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 20:04:22.171420 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.167907    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"]
Nov 03 20:04:22.171420 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.168811    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c18bea-openshift-default-dockercfg-bmhwm\""
Nov 03 20:04:22.222748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.222711    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-plugins-config-volume\") pod \"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3\" (UID: \"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3\") "
Nov 03 20:04:22.223321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.223293    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762a9a66-e013-496b-b744-dc20fe6d4b13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/762a9a66-e013-496b-b744-dc20fe6d4b13-plugins-config-volume\") pod \"762a9a66-e013-496b-b744-dc20fe6d4b13\" (UID: \"762a9a66-e013-496b-b744-dc20fe6d4b13\") "
Nov 03 20:04:22.224155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.224136    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-kube-api-access-27vfr\" (UniqueName: \"kubernetes.io/projected/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-kube-api-access-27vfr\") pod \"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3\" (UID: \"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3\") "
Nov 03 20:04:22.224404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.224386    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762a9a66-e013-496b-b744-dc20fe6d4b13-kube-api-access-hrwzw\" (UniqueName: \"kubernetes.io/projected/762a9a66-e013-496b-b744-dc20fe6d4b13-kube-api-access-hrwzw\") pod \"762a9a66-e013-496b-b744-dc20fe6d4b13\" (UID: \"762a9a66-e013-496b-b744-dc20fe6d4b13\") "
Nov 03 20:04:22.225674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.225649    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-data\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.225942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.223890    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762a9a66-e013-496b-b744-dc20fe6d4b13-plugins-config-volume" pod "762a9a66-e013-496b-b744-dc20fe6d4b13" (UID: "762a9a66-e013-496b-b744-dc20fe6d4b13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:22.226061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.224047    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-plugins-config-volume" pod "e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3" (UID: "e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:22.226544 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.226523    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-envoy\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.226766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.226748    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-token\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.226982 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.226964    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.227234 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.227214    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-credential-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.227612 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.227592    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istiod-ca-cert\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.227825 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.227808    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-podinfo\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.227980 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.227964    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lffdv\" (UniqueName: \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-kube-api-access-lffdv\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.228235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.228218    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-certs\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.236923 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.236897    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:22.237073 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.237059    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/762a9a66-e013-496b-b744-dc20fe6d4b13-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:22.240712 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.240683    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762a9a66-e013-496b-b744-dc20fe6d4b13-kube-api-access-hrwzw" pod "762a9a66-e013-496b-b744-dc20fe6d4b13" (UID: "762a9a66-e013-496b-b744-dc20fe6d4b13"). InnerVolumeSpecName "kube-api-access-hrwzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:22.242805 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.242780    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-kube-api-access-27vfr" pod "e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3" (UID: "e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3"). InnerVolumeSpecName "kube-api-access-27vfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:22.245781 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.245754    4531 generic.go:356] "Generic (PLEG): container finished" podID="e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3" containerID="ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b" exitCode=0
Nov 03 20:04:22.246157 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.246141    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"
Nov 03 20:04:22.249991 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.249959    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb" event={"ID":"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3","Type":"ContainerDied","Data":"ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b"}
Nov 03 20:04:22.250101 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.250030    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb" event={"ID":"e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3","Type":"ContainerDied","Data":"02e5d8ae08128a43b419694435a2233d775bcca719cc62595376159c372eb0b7"}
Nov 03 20:04:22.250101 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.250065    4531 scope.go:122] "RemoveContainer" containerID="ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b"
Nov 03 20:04:22.254679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.254652    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm" event={"ID":"00be0ef0-9ee6-4f14-a738-523b34ebb8bd","Type":"ContainerStarted","Data":"091436b32a26221aae57bc73ad4173b0422f3345ede26daa45b776df03102dce"}
Nov 03 20:04:22.257082 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.257059    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm" event={"ID":"00be0ef0-9ee6-4f14-a738-523b34ebb8bd","Type":"ContainerStarted","Data":"43d3e35df0840bd26f1a390232a2ac218220c05a6cbe513d7ec2b4c696e2268e"}
Nov 03 20:04:22.260026 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.260005    4531 generic.go:356] "Generic (PLEG): container finished" podID="762a9a66-e013-496b-b744-dc20fe6d4b13" containerID="43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a" exitCode=0
Nov 03 20:04:22.261460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.261444    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"
Nov 03 20:04:22.262581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.262554    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4" event={"ID":"762a9a66-e013-496b-b744-dc20fe6d4b13","Type":"ContainerDied","Data":"43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a"}
Nov 03 20:04:22.262676 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.262609    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4" event={"ID":"762a9a66-e013-496b-b744-dc20fe6d4b13","Type":"ContainerDied","Data":"5fdf84f0013cbf73c6c7ac7c5078e7488c59de3ee494b78ae76b26eb5f8fafc5"}
Nov 03 20:04:22.275013 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.274992    4531 scope.go:122] "RemoveContainer" containerID="ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b"
Nov 03 20:04:22.275758 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:22.275619    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b\": container with ID starting with ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b not found: ID does not exist" containerID="ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b"
Nov 03 20:04:22.275758 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.275659    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b"} err="failed to get container status \"ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b\": rpc error: code = NotFound desc = could not find container \"ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b\": container with ID starting with ccefdc3c77a8d15161ae3eb9e78c27b2dbff4530599d5019bf7fe17a8804c89b not found: ID does not exist"
Nov 03 20:04:22.275758 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.275682    4531 scope.go:122] "RemoveContainer" containerID="43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a"
Nov 03 20:04:22.279750 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.279707    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm" podStartSLOduration=2.279690319 podStartE2EDuration="2.279690319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:22.27345951 +0000 UTC m=+1975.543582805" watchObservedRunningTime="2026-11-03 20:04:22.279690319 +0000 UTC m=+1975.549813615"
Nov 03 20:04:22.294545 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.294513    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"]
Nov 03 20:04:22.299902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.299882    4531 scope.go:122] "RemoveContainer" containerID="43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a"
Nov 03 20:04:22.300288 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:22.300261    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a\": container with ID starting with 43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a not found: ID does not exist" containerID="43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a"
Nov 03 20:04:22.300381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.300299    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a"} err="failed to get container status \"43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a\": rpc error: code = NotFound desc = could not find container \"43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a\": container with ID starting with 43aa4bcb304bd85d52d8ddb8c3704497e9ca3ac14546f13233fa8de16cfa925a not found: ID does not exist"
Nov 03 20:04:22.300904 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.300879    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7a62d4-5cdd46ddbc-txvcb"]
Nov 03 20:04:22.313274 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.313166    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"]
Nov 03 20:04:22.322506 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.322464    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7a62d4-89cbcddfc-nhtr4"]
Nov 03 20:04:22.339526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.339469    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istiod-ca-cert\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.339711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.339591    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-podinfo\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.339711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.339624    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffdv\" (UniqueName: \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-kube-api-access-lffdv\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.339851 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.339838    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-certs\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.339946 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.339926    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-data\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.340107 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.340090    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-envoy\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.340226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.340157    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-token\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.340341 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.340317    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.340588 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.340561    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-credential-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.340843 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.340822    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27vfr\" (UniqueName: \"kubernetes.io/projected/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3-kube-api-access-27vfr\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:22.340918 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.340850    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrwzw\" (UniqueName: \"kubernetes.io/projected/762a9a66-e013-496b-b744-dc20fe6d4b13-kube-api-access-hrwzw\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:22.341347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.341321    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-credential-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.342843 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.342805    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istiod-ca-cert\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.344277 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.344256    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.345350 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.345323    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-data\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.345350 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.345338    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-certs\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.352226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.352203    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-envoy\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.357004 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.356983    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-podinfo\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.361636 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.361607    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lffdv\" (UniqueName: \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-kube-api-access-lffdv\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.363931 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.363885    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-token\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.419959 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.419912    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:22.510947 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.510867    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:22.577011 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.576899    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:22.750902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.750707    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:22.826330 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.826252    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"]
Nov 03 20:04:22.936636 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.936604    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762a9a66-e013-496b-b744-dc20fe6d4b13" path="/var/lib/kubelet/pods/762a9a66-e013-496b-b744-dc20fe6d4b13/volumes"
Nov 03 20:04:22.938528 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.937982    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3" path="/var/lib/kubelet/pods/e41a7e6a-0f0c-4d38-ac4a-bb5b534cdda3/volumes"
Nov 03 20:04:22.939238 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:22.939218    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11c1a8-7cd5d"]
Nov 03 20:04:23.053167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:23.045888    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:23.179715 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:23.178722    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"]
Nov 03 20:04:23.196367 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:04:23.196296    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257503a5_c496_4bb6_a0fe_ed4295b9dacf.slice/crio-79dce8458c6b32c7e2ed27b8028aeb7c50a036a55fe0557e1f5ee9fe062702dc WatchSource:0}: Error finding container 79dce8458c6b32c7e2ed27b8028aeb7c50a036a55fe0557e1f5ee9fe062702dc: Status 404 returned error can't find the container with id 79dce8458c6b32c7e2ed27b8028aeb7c50a036a55fe0557e1f5ee9fe062702dc
Nov 03 20:04:23.205061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:23.205021    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:23.205205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:23.205113    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:23.205205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:23.205165    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:23.274539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:23.274504    4531 generic.go:356] "Generic (PLEG): container finished" podID="00be0ef0-9ee6-4f14-a738-523b34ebb8bd" containerID="091436b32a26221aae57bc73ad4173b0422f3345ede26daa45b776df03102dce" exitCode=0
Nov 03 20:04:23.274678 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:23.274614    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm" event={"ID":"00be0ef0-9ee6-4f14-a738-523b34ebb8bd","Type":"ContainerDied","Data":"091436b32a26221aae57bc73ad4173b0422f3345ede26daa45b776df03102dce"}
Nov 03 20:04:23.278682 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:23.278656    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" event={"ID":"257503a5-c496-4bb6-a0fe-ed4295b9dacf","Type":"ContainerStarted","Data":"79dce8458c6b32c7e2ed27b8028aeb7c50a036a55fe0557e1f5ee9fe062702dc"}
Nov 03 20:04:23.284871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:23.284844    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:23.828278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:23.828237    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:24.009096 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.009056    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:24.337196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.333341    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" event={"ID":"257503a5-c496-4bb6-a0fe-ed4295b9dacf","Type":"ContainerStarted","Data":"7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb"}
Nov 03 20:04:24.394951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.394872    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" podStartSLOduration=3.394848896 podStartE2EDuration="3.394848896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:24.372665483 +0000 UTC m=+1977.642788780" watchObservedRunningTime="2026-11-03 20:04:24.394848896 +0000 UTC m=+1977.664972193"
Nov 03 20:04:24.511210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.511143    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:24.515978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.515932    4531 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 20:04:24.516167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.515988    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" podUID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 20:04:24.614071 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.613980    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:24.712041 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.712002    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-xks-ca\") pod \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") "
Nov 03 20:04:24.712370 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.712345    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-kube-api-access-9x8pb\" (UniqueName: \"kubernetes.io/projected/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-kube-api-access-9x8pb\") pod \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") "
Nov 03 20:04:24.712522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.712505    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-openshift-ca\") pod \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\" (UID: \"00be0ef0-9ee6-4f14-a738-523b34ebb8bd\") "
Nov 03 20:04:24.712958 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.712921    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-openshift-ca" pod "00be0ef0-9ee6-4f14-a738-523b34ebb8bd" (UID: "00be0ef0-9ee6-4f14-a738-523b34ebb8bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:24.713415 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.713392    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:24.714487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.714464    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-xks-ca" pod "00be0ef0-9ee6-4f14-a738-523b34ebb8bd" (UID: "00be0ef0-9ee6-4f14-a738-523b34ebb8bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:24.715011 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.714972    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-kube-api-access-9x8pb" pod "00be0ef0-9ee6-4f14-a738-523b34ebb8bd" (UID: "00be0ef0-9ee6-4f14-a738-523b34ebb8bd"). InnerVolumeSpecName "kube-api-access-9x8pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:24.816002 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.815969    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x8pb\" (UniqueName: \"kubernetes.io/projected/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-kube-api-access-9x8pb\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:24.816260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.816238    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00be0ef0-9ee6-4f14-a738-523b34ebb8bd-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:24.916662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.916508    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:24.934654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:24.934621    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e4b9f4-67da-4926-9402-e7f82a5805ec" path="/var/lib/kubelet/pods/e8e4b9f4-67da-4926-9402-e7f82a5805ec/volumes"
Nov 03 20:04:25.365954 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:25.365865    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"
Nov 03 20:04:25.366154 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:25.365879    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm" event={"ID":"00be0ef0-9ee6-4f14-a738-523b34ebb8bd","Type":"ContainerDied","Data":"43d3e35df0840bd26f1a390232a2ac218220c05a6cbe513d7ec2b4c696e2268e"}
Nov 03 20:04:25.366154 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:25.365999    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43d3e35df0840bd26f1a390232a2ac218220c05a6cbe513d7ec2b4c696e2268e"
Nov 03 20:04:25.512107 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:25.512077    4531 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 20:04:25.512305 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:25.512127    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" podUID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 20:04:25.560198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:25.550037    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:26.257204 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:26.254128    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:26.511533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:26.511442    4531 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 20:04:26.511533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:26.511505    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" podUID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 20:04:27.053670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:27.053595    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:27.511546 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:27.511509    4531 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 20:04:27.511969 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:27.511563    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" podUID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 20:04:27.727861 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:27.727787    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:28.411199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:28.395875    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:28.512605 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:28.511725    4531 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 20:04:28.512605 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:28.511791    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" podUID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 20:04:29.261113 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:29.260731    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:29.470198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:29.465092    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:29.593867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:29.593773    4531 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 20:04:29.594485 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:29.594451    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" podUID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 20:04:29.687195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:29.649240    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:29.687195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:29.673311    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"]
Nov 03 20:04:29.766312 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:29.764216    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"]
Nov 03 20:04:29.766312 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:29.764502    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerName="istio-proxy" containerID="cri-o://1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf" gracePeriod=30
Nov 03 20:04:29.885254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:29.859327    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy" containerID="cri-o://8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369" gracePeriod=30
Nov 03 20:04:29.981410 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:29.980230    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"]
Nov 03 20:04:29.981410 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:29.980494    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" containerName="istio-proxy" containerID="cri-o://48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc" gracePeriod=30
Nov 03 20:04:30.089259 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.086458    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:30.097499 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.096483    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"]
Nov 03 20:04:30.132843 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.132798    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"]
Nov 03 20:04:30.134667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.134624    4531 cpu_manager.go:414] "containerMap: removing container" podUID="762a9a66-e013-496b-b744-dc20fe6d4b13" containerName="payload-processing"
Nov 03 20:04:30.134667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.134658    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762a9a66-e013-496b-b744-dc20fe6d4b13" containerName="payload-processing"
Nov 03 20:04:30.134853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.134733    4531 cpu_manager.go:414] "containerMap: removing container" podUID="00be0ef0-9ee6-4f14-a738-523b34ebb8bd" containerName="revoke-keys"
Nov 03 20:04:30.134853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.134742    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00be0ef0-9ee6-4f14-a738-523b34ebb8bd" containerName="revoke-keys"
Nov 03 20:04:30.134945 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.134931    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="00be0ef0-9ee6-4f14-a738-523b34ebb8bd" containerName="revoke-keys"
Nov 03 20:04:30.134986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.134958    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="762a9a66-e013-496b-b744-dc20fe6d4b13" containerName="payload-processing"
Nov 03 20:04:30.141678 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.141616    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"]
Nov 03 20:04:30.141910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.141744    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:30.216850 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.216822    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:04:30.217210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.217142    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:04:30.261019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.260410    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:30.261019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.260515    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrzf\" (UniqueName: \"kubernetes.io/projected/c38b4b32-93b2-4b98-9233-02442a28118d-kube-api-access-dwrzf\") pod \"maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:30.261019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.260646    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:30.362953 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.362912    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:30.363207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.363185    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:30.363301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.363251    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrzf\" (UniqueName: \"kubernetes.io/projected/c38b4b32-93b2-4b98-9233-02442a28118d-kube-api-access-dwrzf\") pod \"maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:30.363976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.363948    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:30.365418 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.365393    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:30.375876 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.375848    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrzf\" (UniqueName: \"kubernetes.io/projected/c38b4b32-93b2-4b98-9233-02442a28118d-kube-api-access-dwrzf\") pod \"maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:30.464280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.464131    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:30.669102 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.669066    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:30.708224 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.698565    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:30.708224 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.700392    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:30.746504 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.734064    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:04:30.803135 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:30.802869    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"]
Nov 03 20:04:31.587198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:31.587090    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h" event={"ID":"c38b4b32-93b2-4b98-9233-02442a28118d","Type":"ContainerStarted","Data":"7f3629ba7b7d3ccec38f2f4dfca22feb207b4555ee9aa240ee5bec79533d8945"}
Nov 03 20:04:31.587198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:31.587150    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h" event={"ID":"c38b4b32-93b2-4b98-9233-02442a28118d","Type":"ContainerStarted","Data":"dcc8474b7fa0c7869b52fd194adcba034d86e0ea0e6e9992b31146caf09f0f63"}
Nov 03 20:04:31.601698 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:31.601640    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h" podStartSLOduration=1.601624967 podStartE2EDuration="1.601624967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:31.60143721 +0000 UTC m=+1984.871560507" watchObservedRunningTime="2026-11-03 20:04:31.601624967 +0000 UTC m=+1984.871748262"
Nov 03 20:04:31.659206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:31.656715    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:32.529205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:32.529136    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:32.609145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:32.609109    4531 generic.go:356] "Generic (PLEG): container finished" podID="c38b4b32-93b2-4b98-9233-02442a28118d" containerID="7f3629ba7b7d3ccec38f2f4dfca22feb207b4555ee9aa240ee5bec79533d8945" exitCode=0
Nov 03 20:04:32.609355 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:32.609232    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h" event={"ID":"c38b4b32-93b2-4b98-9233-02442a28118d","Type":"ContainerDied","Data":"7f3629ba7b7d3ccec38f2f4dfca22feb207b4555ee9aa240ee5bec79533d8945"}
Nov 03 20:04:32.993000 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:32.992964    4531 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:04:32.993201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:32.993024    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:04:33.362059 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:33.361938    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:33.793990 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:33.793963    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:33.928009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:33.927975    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:33.962195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:33.950730    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-openshift-ca\") pod \"c38b4b32-93b2-4b98-9233-02442a28118d\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") "
Nov 03 20:04:33.962195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:33.950806    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-xks-ca\") pod \"c38b4b32-93b2-4b98-9233-02442a28118d\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") "
Nov 03 20:04:33.962195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:33.951047    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c38b4b32-93b2-4b98-9233-02442a28118d-kube-api-access-dwrzf\" (UniqueName: \"kubernetes.io/projected/c38b4b32-93b2-4b98-9233-02442a28118d-kube-api-access-dwrzf\") pod \"c38b4b32-93b2-4b98-9233-02442a28118d\" (UID: \"c38b4b32-93b2-4b98-9233-02442a28118d\") "
Nov 03 20:04:33.990411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:33.975445    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38b4b32-93b2-4b98-9233-02442a28118d-kube-api-access-dwrzf" pod "c38b4b32-93b2-4b98-9233-02442a28118d" (UID: "c38b4b32-93b2-4b98-9233-02442a28118d"). InnerVolumeSpecName "kube-api-access-dwrzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:33.990411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:33.975639    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-openshift-ca" pod "c38b4b32-93b2-4b98-9233-02442a28118d" (UID: "c38b4b32-93b2-4b98-9233-02442a28118d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:33.990411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:33.978647    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:33.990411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:33.978673    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwrzf\" (UniqueName: \"kubernetes.io/projected/c38b4b32-93b2-4b98-9233-02442a28118d-kube-api-access-dwrzf\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:33.996206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:33.995868    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-xks-ca" pod "c38b4b32-93b2-4b98-9233-02442a28118d" (UID: "c38b4b32-93b2-4b98-9233-02442a28118d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:34.080558 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.080524    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38b4b32-93b2-4b98-9233-02442a28118d-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:34.359531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.359428    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"]
Nov 03 20:04:34.361413 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.361372    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c38b4b32-93b2-4b98-9233-02442a28118d" containerName="revoke-keys"
Nov 03 20:04:34.361413 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.361411    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38b4b32-93b2-4b98-9233-02442a28118d" containerName="revoke-keys"
Nov 03 20:04:34.361840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.361821    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38b4b32-93b2-4b98-9233-02442a28118d" containerName="revoke-keys"
Nov 03 20:04:34.370065 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.370035    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.375007 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.374537    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"]
Nov 03 20:04:34.376232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.376204    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-9f3188-openshift-default-dockercfg-xp46f\""
Nov 03 20:04:34.490616 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.490579    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.490776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.490624    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.490838 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.490791    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.490838 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.490826    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.490944 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.490931    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.490991 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.490979    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrgns\" (UniqueName: \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-kube-api-access-hrgns\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.491120 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.491105    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.491251 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.491234    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.491323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.491291    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.593370 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.593328    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.593370 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.593383    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.593669 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.593515    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.593669 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.593544    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.593669 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.593625    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.593857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.593741    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrgns\" (UniqueName: \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-kube-api-access-hrgns\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.593924 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.593853    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.593982 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.593912    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.594033 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.593981    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.594093 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.594036    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.594307 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.594281    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.594410 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.594372    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.594547 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.594531    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.594615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.594564    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.596223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.596162    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.596370 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.596353    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.600851 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.600830    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.600851 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.600842    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrgns\" (UniqueName: \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-kube-api-access-hrgns\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.631747 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.631715    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"
Nov 03 20:04:34.631929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.631753    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h" event={"ID":"c38b4b32-93b2-4b98-9233-02442a28118d","Type":"ContainerDied","Data":"dcc8474b7fa0c7869b52fd194adcba034d86e0ea0e6e9992b31146caf09f0f63"}
Nov 03 20:04:34.631929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.631813    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcc8474b7fa0c7869b52fd194adcba034d86e0ea0e6e9992b31146caf09f0f63"
Nov 03 20:04:34.698258 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.698225    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:34.737230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:34.737114    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:35.080558 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.080328    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"]
Nov 03 20:04:35.089923 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:04:35.087404    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edf055d_9f1e_415e_a672_8cfd8e4d2263.slice/crio-7dbd35c44d001f180e3a794cd7b92a1d364200b766b9d005c29aaa3d848f6c2f WatchSource:0}: Error finding container 7dbd35c44d001f180e3a794cd7b92a1d364200b766b9d005c29aaa3d848f6c2f: Status 404 returned error can't find the container with id 7dbd35c44d001f180e3a794cd7b92a1d364200b766b9d005c29aaa3d848f6c2f
Nov 03 20:04:35.102318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.102270    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:35.102422 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.102382    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:35.102491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.102437    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:35.273703 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.272596    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:35.322433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.320538    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-socket\") pod \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") "
Nov 03 20:04:35.322433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.320630    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-kube-api-access-7bnkg\" (UniqueName: \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-kube-api-access-7bnkg\") pod \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") "
Nov 03 20:04:35.322433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.320669    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-envoy\") pod \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") "
Nov 03 20:04:35.322433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.320796    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-certs\") pod \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") "
Nov 03 20:04:35.322433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.320902    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istiod-ca-cert\") pod \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") "
Nov 03 20:04:35.322433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.320939    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-credential-socket\") pod \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") "
Nov 03 20:04:35.322433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.321000    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-token\" (UniqueName: \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-token\") pod \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") "
Nov 03 20:04:35.322433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.321106    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-podinfo\") pod \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") "
Nov 03 20:04:35.322433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.321292    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-data\") pod \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\" (UID: \"595c6bfd-72b7-4501-95ce-5c7e3eda018c\") "
Nov 03 20:04:35.323375 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.323161    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-data" pod "595c6bfd-72b7-4501-95ce-5c7e3eda018c" (UID: "595c6bfd-72b7-4501-95ce-5c7e3eda018c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.324944 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.324462    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-socket" pod "595c6bfd-72b7-4501-95ce-5c7e3eda018c" (UID: "595c6bfd-72b7-4501-95ce-5c7e3eda018c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.327348 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.326207    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-certs" pod "595c6bfd-72b7-4501-95ce-5c7e3eda018c" (UID: "595c6bfd-72b7-4501-95ce-5c7e3eda018c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.327553 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.327518    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-credential-socket" pod "595c6bfd-72b7-4501-95ce-5c7e3eda018c" (UID: "595c6bfd-72b7-4501-95ce-5c7e3eda018c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.327986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.327964    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istiod-ca-cert" pod "595c6bfd-72b7-4501-95ce-5c7e3eda018c" (UID: "595c6bfd-72b7-4501-95ce-5c7e3eda018c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:35.329526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.329497    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-podinfo" pod "595c6bfd-72b7-4501-95ce-5c7e3eda018c" (UID: "595c6bfd-72b7-4501-95ce-5c7e3eda018c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:04:35.330305 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.330276    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-kube-api-access-7bnkg" pod "595c6bfd-72b7-4501-95ce-5c7e3eda018c" (UID: "595c6bfd-72b7-4501-95ce-5c7e3eda018c"). InnerVolumeSpecName "kube-api-access-7bnkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.333330 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.333218    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-envoy" pod "595c6bfd-72b7-4501-95ce-5c7e3eda018c" (UID: "595c6bfd-72b7-4501-95ce-5c7e3eda018c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.339219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.338357    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 20:04:35.339956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.339925    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-token" pod "595c6bfd-72b7-4501-95ce-5c7e3eda018c" (UID: "595c6bfd-72b7-4501-95ce-5c7e3eda018c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.423914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.423858    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-credential-socket\") pod \"a2400a16-c022-45cb-91ac-8fb743616e13\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") "
Nov 03 20:04:35.424130 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.424023    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-envoy\") pod \"a2400a16-c022-45cb-91ac-8fb743616e13\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") "
Nov 03 20:04:35.424245 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.424138    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2400a16-c022-45cb-91ac-8fb743616e13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2400a16-c022-45cb-91ac-8fb743616e13-istio-podinfo\") pod \"a2400a16-c022-45cb-91ac-8fb743616e13\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") "
Nov 03 20:04:35.424325 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.424271    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-certs\") pod \"a2400a16-c022-45cb-91ac-8fb743616e13\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") "
Nov 03 20:04:35.424386 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.424347    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-kube-api-access-bpzsz\" (UniqueName: \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-kube-api-access-bpzsz\") pod \"a2400a16-c022-45cb-91ac-8fb743616e13\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") "
Nov 03 20:04:35.424457 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.424404    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-socket\") pod \"a2400a16-c022-45cb-91ac-8fb743616e13\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") "
Nov 03 20:04:35.424513 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.424492    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2400a16-c022-45cb-91ac-8fb743616e13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2400a16-c022-45cb-91ac-8fb743616e13-istiod-ca-cert\") pod \"a2400a16-c022-45cb-91ac-8fb743616e13\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") "
Nov 03 20:04:35.424574 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.424565    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-data\") pod \"a2400a16-c022-45cb-91ac-8fb743616e13\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") "
Nov 03 20:04:35.425048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.424782    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-istio-token\" (UniqueName: \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-istio-token\") pod \"a2400a16-c022-45cb-91ac-8fb743616e13\" (UID: \"a2400a16-c022-45cb-91ac-8fb743616e13\") "
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.425281    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-data" pod "a2400a16-c022-45cb-91ac-8fb743616e13" (UID: "a2400a16-c022-45cb-91ac-8fb743616e13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.425727    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-certs" pod "a2400a16-c022-45cb-91ac-8fb743616e13" (UID: "a2400a16-c022-45cb-91ac-8fb743616e13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.425738    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-socket" pod "a2400a16-c022-45cb-91ac-8fb743616e13" (UID: "a2400a16-c022-45cb-91ac-8fb743616e13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.425888    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2400a16-c022-45cb-91ac-8fb743616e13-istiod-ca-cert" pod "a2400a16-c022-45cb-91ac-8fb743616e13" (UID: "a2400a16-c022-45cb-91ac-8fb743616e13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426201    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426226    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426242    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426256    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426271    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426285    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426298    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2400a16-c022-45cb-91ac-8fb743616e13-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426314    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426328    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426342    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426356    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bnkg\" (UniqueName: \"kubernetes.io/projected/595c6bfd-72b7-4501-95ce-5c7e3eda018c-kube-api-access-7bnkg\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426370    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.426426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.426384    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595c6bfd-72b7-4501-95ce-5c7e3eda018c-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.427887 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.427849    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-credential-socket" pod "a2400a16-c022-45cb-91ac-8fb743616e13" (UID: "a2400a16-c022-45cb-91ac-8fb743616e13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.428356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.428258    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-kube-api-access-bpzsz" pod "a2400a16-c022-45cb-91ac-8fb743616e13" (UID: "a2400a16-c022-45cb-91ac-8fb743616e13"). InnerVolumeSpecName "kube-api-access-bpzsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.428356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.428333    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-envoy" pod "a2400a16-c022-45cb-91ac-8fb743616e13" (UID: "a2400a16-c022-45cb-91ac-8fb743616e13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.429118 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.429097    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-istio-token" pod "a2400a16-c022-45cb-91ac-8fb743616e13" (UID: "a2400a16-c022-45cb-91ac-8fb743616e13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.430928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.430907    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2400a16-c022-45cb-91ac-8fb743616e13-istio-podinfo" pod "a2400a16-c022-45cb-91ac-8fb743616e13" (UID: "a2400a16-c022-45cb-91ac-8fb743616e13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:04:35.471209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.471165    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 20:04:35.527144 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.527110    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-kube-api-access-r6jwf\" (UniqueName: \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-kube-api-access-r6jwf\") pod \"e6364721-945b-4bfc-b118-cf93b16619d3\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") "
Nov 03 20:04:35.527144 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.527149    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-istio-token\" (UniqueName: \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-istio-token\") pod \"e6364721-945b-4bfc-b118-cf93b16619d3\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") "
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.528292    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6364721-945b-4bfc-b118-cf93b16619d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6364721-945b-4bfc-b118-cf93b16619d3-istio-podinfo\") pod \"e6364721-945b-4bfc-b118-cf93b16619d3\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") "
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.528364    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-envoy\") pod \"e6364721-945b-4bfc-b118-cf93b16619d3\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") "
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.528485    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6364721-945b-4bfc-b118-cf93b16619d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6364721-945b-4bfc-b118-cf93b16619d3-istiod-ca-cert\") pod \"e6364721-945b-4bfc-b118-cf93b16619d3\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") "
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.528523    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-data\") pod \"e6364721-945b-4bfc-b118-cf93b16619d3\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") "
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.528575    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-certs\") pod \"e6364721-945b-4bfc-b118-cf93b16619d3\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") "
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.528630    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-credential-socket\") pod \"e6364721-945b-4bfc-b118-cf93b16619d3\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") "
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.528670    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-socket\") pod \"e6364721-945b-4bfc-b118-cf93b16619d3\" (UID: \"e6364721-945b-4bfc-b118-cf93b16619d3\") "
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.529555    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-data" pod "e6364721-945b-4bfc-b118-cf93b16619d3" (UID: "e6364721-945b-4bfc-b118-cf93b16619d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.529629    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6364721-945b-4bfc-b118-cf93b16619d3-istiod-ca-cert" pod "e6364721-945b-4bfc-b118-cf93b16619d3" (UID: "e6364721-945b-4bfc-b118-cf93b16619d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.529654    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.529680    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2400a16-c022-45cb-91ac-8fb743616e13-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.529693    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2400a16-c022-45cb-91ac-8fb743616e13-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.529707    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpzsz\" (UniqueName: \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-kube-api-access-bpzsz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.529720    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2400a16-c022-45cb-91ac-8fb743616e13-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.530282    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-certs" pod "e6364721-945b-4bfc-b118-cf93b16619d3" (UID: "e6364721-945b-4bfc-b118-cf93b16619d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.530370    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-socket" pod "e6364721-945b-4bfc-b118-cf93b16619d3" (UID: "e6364721-945b-4bfc-b118-cf93b16619d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.551455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.549520    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-istio-token" pod "e6364721-945b-4bfc-b118-cf93b16619d3" (UID: "e6364721-945b-4bfc-b118-cf93b16619d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.552659 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.552598    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-credential-socket" pod "e6364721-945b-4bfc-b118-cf93b16619d3" (UID: "e6364721-945b-4bfc-b118-cf93b16619d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.573227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.563466    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-kube-api-access-r6jwf" pod "e6364721-945b-4bfc-b118-cf93b16619d3" (UID: "e6364721-945b-4bfc-b118-cf93b16619d3"). InnerVolumeSpecName "kube-api-access-r6jwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.646012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.645256    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6364721-945b-4bfc-b118-cf93b16619d3-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.646012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.645290    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.646012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.645307    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.646012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.645320    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.646012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.645334    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.646012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.645348    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6jwf\" (UniqueName: \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-kube-api-access-r6jwf\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.646012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.645363    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6364721-945b-4bfc-b118-cf93b16619d3-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.780451 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.775156    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6364721-945b-4bfc-b118-cf93b16619d3-istio-podinfo" pod "e6364721-945b-4bfc-b118-cf93b16619d3" (UID: "e6364721-945b-4bfc-b118-cf93b16619d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:04:35.810197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.791244    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6364721-945b-4bfc-b118-cf93b16619d3-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.810197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.792872    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-envoy" pod "e6364721-945b-4bfc-b118-cf93b16619d3" (UID: "e6364721-945b-4bfc-b118-cf93b16619d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.819301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.812490    4531 generic.go:356] "Generic (PLEG): container finished" podID="e6364721-945b-4bfc-b118-cf93b16619d3" containerID="48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc" exitCode=0
Nov 03 20:04:35.819301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.812588    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" event={"ID":"e6364721-945b-4bfc-b118-cf93b16619d3","Type":"ContainerDied","Data":"48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc"}
Nov 03 20:04:35.819301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.812638    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m" event={"ID":"e6364721-945b-4bfc-b118-cf93b16619d3","Type":"ContainerDied","Data":"ce1599e9d4bb283db639d3e730289b32eb8fbbbff3c3707cbacf06240c13b381"}
Nov 03 20:04:35.819301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.812668    4531 scope.go:122] "RemoveContainer" containerID="48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc"
Nov 03 20:04:35.819301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.812836    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"
Nov 03 20:04:35.833045 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.833003    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" event={"ID":"9edf055d-9f1e-415e-a672-8cfd8e4d2263","Type":"ContainerStarted","Data":"56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb"}
Nov 03 20:04:35.833449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.833072    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" event={"ID":"9edf055d-9f1e-415e-a672-8cfd8e4d2263","Type":"ContainerStarted","Data":"7dbd35c44d001f180e3a794cd7b92a1d364200b766b9d005c29aaa3d848f6c2f"}
Nov 03 20:04:35.849396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.842489    4531 generic.go:356] "Generic (PLEG): container finished" podID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerID="1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf" exitCode=0
Nov 03 20:04:35.849396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.842668    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" event={"ID":"595c6bfd-72b7-4501-95ce-5c7e3eda018c","Type":"ContainerDied","Data":"1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf"}
Nov 03 20:04:35.849396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.842721    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t" event={"ID":"595c6bfd-72b7-4501-95ce-5c7e3eda018c","Type":"ContainerDied","Data":"ecdd42645376188cb8e5be67ccdcaa20d463f9ebabbf6f39b261589588da4c01"}
Nov 03 20:04:35.849396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.842803    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"
Nov 03 20:04:35.849396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.844587    4531 generic.go:356] "Generic (PLEG): container finished" podID="a2400a16-c022-45cb-91ac-8fb743616e13" containerID="8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369" exitCode=0
Nov 03 20:04:35.849396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.844616    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" event={"ID":"a2400a16-c022-45cb-91ac-8fb743616e13","Type":"ContainerDied","Data":"8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369"}
Nov 03 20:04:35.849396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.844647    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l" event={"ID":"a2400a16-c022-45cb-91ac-8fb743616e13","Type":"ContainerDied","Data":"e0d0c301520c9bcd0921e0615012e846577d22cd35cb42c7d9908d76984feae7"}
Nov 03 20:04:35.849396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.844700    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"
Nov 03 20:04:35.852973 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.852630    4531 scope.go:122] "RemoveContainer" containerID="48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc"
Nov 03 20:04:35.852973 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:35.852915    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc\": container with ID starting with 48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc not found: ID does not exist" containerID="48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc"
Nov 03 20:04:35.852973 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.852950    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc"} err="failed to get container status \"48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc\": rpc error: code = NotFound desc = could not find container \"48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc\": container with ID starting with 48ac6c1e1073d576b8e897fbc4bdce0c5188f93485bee49ca4b3eb5ca8cbeadc not found: ID does not exist"
Nov 03 20:04:35.853150 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.852977    4531 scope.go:122] "RemoveContainer" containerID="1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf"
Nov 03 20:04:35.867710 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.867343    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" podStartSLOduration=1.867324912 podStartE2EDuration="1.867324912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:35.856568807 +0000 UTC m=+1989.126692103" watchObservedRunningTime="2026-11-03 20:04:35.867324912 +0000 UTC m=+1989.137448208"
Nov 03 20:04:35.868953 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.868930    4531 scope.go:122] "RemoveContainer" containerID="1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf"
Nov 03 20:04:35.876715 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:35.876225    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf\": container with ID starting with 1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf not found: ID does not exist" containerID="1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf"
Nov 03 20:04:35.876715 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.876271    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf"} err="failed to get container status \"1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf\": rpc error: code = NotFound desc = could not find container \"1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf\": container with ID starting with 1871e7ed3b88ac6e7d3f7034bb414fb0d32d73aba9dc45d1e928bfcffbbe9adf not found: ID does not exist"
Nov 03 20:04:35.876715 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.876313    4531 scope.go:122] "RemoveContainer" containerID="8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369"
Nov 03 20:04:35.886320 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.886293    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"]
Nov 03 20:04:35.890936 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.890918    4531 scope.go:122] "RemoveContainer" containerID="8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369"
Nov 03 20:04:35.893072 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.893049    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6364721-945b-4bfc-b118-cf93b16619d3-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.893313 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:35.893278    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369\": container with ID starting with 8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369 not found: ID does not exist" containerID="8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369"
Nov 03 20:04:35.893418 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.893317    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369"} err="failed to get container status \"8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369\": rpc error: code = NotFound desc = could not find container \"8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369\": container with ID starting with 8e2efc85ea16b88c9bac17c72ef31a4996611ece7afa820a3c241aa248746369 not found: ID does not exist"
Nov 03 20:04:35.893724 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.893704    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-k2k5m"]
Nov 03 20:04:35.900073 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.900047    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"]
Nov 03 20:04:35.904939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.904915    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-pqh5t"]
Nov 03 20:04:35.916301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.916273    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"]
Nov 03 20:04:35.938528 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.922583    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wcn4l"]
Nov 03 20:04:35.946910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:35.946879    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:36.110912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:36.110830    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:36.699812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:36.699778    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:36.701264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:36.701236    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 20:04:36.701408 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:36.701294    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 20:04:36.885441 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:36.885402    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" path="/var/lib/kubelet/pods/595c6bfd-72b7-4501-95ce-5c7e3eda018c/volumes"
Nov 03 20:04:36.886347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:36.886320    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" path="/var/lib/kubelet/pods/a2400a16-c022-45cb-91ac-8fb743616e13/volumes"
Nov 03 20:04:36.887252 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:36.887229    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" path="/var/lib/kubelet/pods/e6364721-945b-4bfc-b118-cf93b16619d3/volumes"
Nov 03 20:04:36.909074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:36.909040    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p" podUID="8fa5ca17-a405-43b0-bf42-acdf79ff1a70" containerName="main" containerID="cri-o://009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508" gracePeriod=45
Nov 03 20:04:36.962323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:36.962204    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p" podUID="8fa5ca17-a405-43b0-bf42-acdf79ff1a70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.235:8000/ready\": dial tcp 10.133.0.235:8000: connect: connection refused"
Nov 03 20:04:37.062845 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.062781    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:37.364938 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.364902    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:04:37.425386 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.425356    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-home\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-home\") pod \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") "
Nov 03 20:04:37.425593 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.425417    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-kube-api-access-ngmhq\" (UniqueName: \"kubernetes.io/projected/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-kube-api-access-ngmhq\") pod \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") "
Nov 03 20:04:37.425593 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.425499    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-model-cache\") pod \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") "
Nov 03 20:04:37.425593 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.425585    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tmp-dir\") pod \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") "
Nov 03 20:04:37.425788 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.425642    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-dshm\") pod \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") "
Nov 03 20:04:37.425788 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.425652    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-home" pod "8fa5ca17-a405-43b0-bf42-acdf79ff1a70" (UID: "8fa5ca17-a405-43b0-bf42-acdf79ff1a70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:37.425788 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.425693    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-model-cache" pod "8fa5ca17-a405-43b0-bf42-acdf79ff1a70" (UID: "8fa5ca17-a405-43b0-bf42-acdf79ff1a70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:37.425788 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.425756    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tls-certs\") pod \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\" (UID: \"8fa5ca17-a405-43b0-bf42-acdf79ff1a70\") "
Nov 03 20:04:37.425971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.425875    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tmp-dir" pod "8fa5ca17-a405-43b0-bf42-acdf79ff1a70" (UID: "8fa5ca17-a405-43b0-bf42-acdf79ff1a70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:37.426822 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.426793    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:37.426934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.426827    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:37.426934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.426842    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:37.427944 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.427921    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-dshm" pod "8fa5ca17-a405-43b0-bf42-acdf79ff1a70" (UID: "8fa5ca17-a405-43b0-bf42-acdf79ff1a70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:37.428043 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.427927    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-kube-api-access-ngmhq" pod "8fa5ca17-a405-43b0-bf42-acdf79ff1a70" (UID: "8fa5ca17-a405-43b0-bf42-acdf79ff1a70"). InnerVolumeSpecName "kube-api-access-ngmhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:37.428097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.428074    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tls-certs" pod "8fa5ca17-a405-43b0-bf42-acdf79ff1a70" (UID: "8fa5ca17-a405-43b0-bf42-acdf79ff1a70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:37.528898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.528864    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:37.528898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.528897    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:37.528898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.528907    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngmhq\" (UniqueName: \"kubernetes.io/projected/8fa5ca17-a405-43b0-bf42-acdf79ff1a70-kube-api-access-ngmhq\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:37.698942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.698900    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 20:04:37.699106 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.698959    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 20:04:37.895092 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.895058    4531 generic.go:356] "Generic (PLEG): container finished" podID="8fa5ca17-a405-43b0-bf42-acdf79ff1a70" containerID="009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508" exitCode=0
Nov 03 20:04:37.895332 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.895103    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"
Nov 03 20:04:37.895332 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.895141    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p" event={"ID":"8fa5ca17-a405-43b0-bf42-acdf79ff1a70","Type":"ContainerDied","Data":"009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508"}
Nov 03 20:04:37.895332 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.895210    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p" event={"ID":"8fa5ca17-a405-43b0-bf42-acdf79ff1a70","Type":"ContainerDied","Data":"21ab63c39b24cd0112f2796f4b3e550f7d979fcedfb8c41fee9637daa329feda"}
Nov 03 20:04:37.895332 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.895253    4531 scope.go:122] "RemoveContainer" containerID="009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508"
Nov 03 20:04:37.908461 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.908442    4531 scope.go:122] "RemoveContainer" containerID="009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508"
Nov 03 20:04:37.908731 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:37.908711    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508\": container with ID starting with 009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508 not found: ID does not exist" containerID="009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508"
Nov 03 20:04:37.908773 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.908739    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508"} err="failed to get container status \"009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508\": rpc error: code = NotFound desc = could not find container \"009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508\": container with ID starting with 009e18efddb04c0896304d8996854283c53286fe624bdbda545e30fd00738508 not found: ID does not exist"
Nov 03 20:04:37.919591 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.919563    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"]
Nov 03 20:04:37.924200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:37.924159    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f3188/auth-test-model-9f3188-kserve-57cfc6cfc-zcw9p"]
Nov 03 20:04:38.060044 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:38.059936    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:38.698847 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:38.698808    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 20:04:38.699252 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:38.698864    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 20:04:38.761317 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:38.758501    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:38.883201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:38.882821    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa5ca17-a405-43b0-bf42-acdf79ff1a70" path="/var/lib/kubelet/pods/8fa5ca17-a405-43b0-bf42-acdf79ff1a70/volumes"
Nov 03 20:04:39.195369 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:39.195327    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:39.354239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:39.342305    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:39.635190 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:39.633322    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:39.700195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:39.699329    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 20:04:39.700195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:39.699393    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 20:04:39.807141 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:39.807094    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:40.061258 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:40.053079    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:40.305351 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:40.302894    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:40.457547 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:40.457507    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"]
Nov 03 20:04:40.468225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:40.462733    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:40.479230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:40.479201    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"]
Nov 03 20:04:40.709241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:40.704194    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 20:04:40.709241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:40.704276    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 20:04:40.720199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:40.716113    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:41.169346 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:41.155422    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:41.498322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:41.472654    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:41.561394 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:41.561340    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"]
Nov 03 20:04:41.562021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:41.561975    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv" podUID="90e841f9-77b1-4f9e-bf90-a444d25f01d7" containerName="maas-api" containerID="cri-o://9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc" gracePeriod=30
Nov 03 20:04:41.662197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:41.658293    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:41.720196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:41.700909    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 20:04:41.720196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:41.700979    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 20:04:41.821206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:41.816630    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:41.840898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:41.840414    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:41.900820 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:41.900770    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:41.984486 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:41.984400    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:42.162197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.153989    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:42.362321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.358660    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 20:04:42.408336 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.408299    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:42.410889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.410857    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-maas-api-tls\") pod \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") "
Nov 03 20:04:42.411082 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.411012    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e841f9-77b1-4f9e-bf90-a444d25f01d7-kube-api-access-xrvd8\" (UniqueName: \"kubernetes.io/projected/90e841f9-77b1-4f9e-bf90-a444d25f01d7-kube-api-access-xrvd8\") pod \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") "
Nov 03 20:04:42.411158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.411089    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e841f9-77b1-4f9e-bf90-a444d25f01d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e841f9-77b1-4f9e-bf90-a444d25f01d7-openshift-service-ca\") pod \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") "
Nov 03 20:04:42.411158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.411111    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-metrics-certs\") pod \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\" (UID: \"90e841f9-77b1-4f9e-bf90-a444d25f01d7\") "
Nov 03 20:04:42.412008 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.411976    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e841f9-77b1-4f9e-bf90-a444d25f01d7-openshift-service-ca" pod "90e841f9-77b1-4f9e-bf90-a444d25f01d7" (UID: "90e841f9-77b1-4f9e-bf90-a444d25f01d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:42.414093 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.414025    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e841f9-77b1-4f9e-bf90-a444d25f01d7-kube-api-access-xrvd8" pod "90e841f9-77b1-4f9e-bf90-a444d25f01d7" (UID: "90e841f9-77b1-4f9e-bf90-a444d25f01d7"). InnerVolumeSpecName "kube-api-access-xrvd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:42.414394 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.414376    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-maas-api-tls" pod "90e841f9-77b1-4f9e-bf90-a444d25f01d7" (UID: "90e841f9-77b1-4f9e-bf90-a444d25f01d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:42.425226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.415674    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-metrics-certs" pod "90e841f9-77b1-4f9e-bf90-a444d25f01d7" (UID: "90e841f9-77b1-4f9e-bf90-a444d25f01d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:42.513041 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.512997    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:42.513041 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.513033    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrvd8\" (UniqueName: \"kubernetes.io/projected/90e841f9-77b1-4f9e-bf90-a444d25f01d7-kube-api-access-xrvd8\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:42.513354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.513048    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e841f9-77b1-4f9e-bf90-a444d25f01d7-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:42.513354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.513062    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90e841f9-77b1-4f9e-bf90-a444d25f01d7-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:42.699315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.699228    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 03 20:04:42.699315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.699289    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 03 20:04:42.796475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.796441    4531 generic.go:356] "Generic (PLEG): container finished" podID="90e841f9-77b1-4f9e-bf90-a444d25f01d7" containerID="9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc" exitCode=0
Nov 03 20:04:42.796940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.796494    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"
Nov 03 20:04:42.796940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.796524    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv" event={"ID":"90e841f9-77b1-4f9e-bf90-a444d25f01d7","Type":"ContainerDied","Data":"9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc"}
Nov 03 20:04:42.796940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.796585    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv" event={"ID":"90e841f9-77b1-4f9e-bf90-a444d25f01d7","Type":"ContainerDied","Data":"e65d585b0bbb030bc3eaaf5c0be260c5d6bd24e8bf570fa39abd3eb595695c2a"}
Nov 03 20:04:42.796940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.796615    4531 scope.go:122] "RemoveContainer" containerID="9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc"
Nov 03 20:04:42.812924 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.812900    4531 scope.go:122] "RemoveContainer" containerID="9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc"
Nov 03 20:04:42.813350 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:42.813327    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc\": container with ID starting with 9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc not found: ID does not exist" containerID="9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc"
Nov 03 20:04:42.813442 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.813359    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc"} err="failed to get container status \"9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc\": rpc error: code = NotFound desc = could not find container \"9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc\": container with ID starting with 9f94e90d75bd99b8a784f5c67ab39db445369f851ab60ebb95fadb20ce04b3cc not found: ID does not exist"
Nov 03 20:04:42.824683 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.824652    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"]
Nov 03 20:04:42.832393 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.832364    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9f3188-5cf5bf4c8d-pgzlv"]
Nov 03 20:04:42.862899 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:42.862867    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e841f9-77b1-4f9e-bf90-a444d25f01d7" path="/var/lib/kubelet/pods/90e841f9-77b1-4f9e-bf90-a444d25f01d7/volumes"
Nov 03 20:04:43.086502 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:43.084980    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:43.754624 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:43.754597    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:43.755131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:43.755110    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:43.755926 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:43.755906    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:43.832001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:43.831947    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:44.539207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:44.537832    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:45.127815 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.127773    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc" podUID="79a95827-f3b5-4c1d-8c1f-419924011fe6" containerName="main" containerID="cri-o://4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084" gracePeriod=45
Nov 03 20:04:45.159230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.145770    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:45.425319 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.425287    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:04:45.458905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.458863    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq" podUID="01880166-1848-4cae-a23b-8ffc842ed06c" containerName="payload-processing" containerID="cri-o://ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b" gracePeriod=25
Nov 03 20:04:45.480648 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.480613    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r" podUID="73377b59-5b9c-4cb9-8c69-413b5c215d30" containerName="payload-pre-processing" containerID="cri-o://ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f" gracePeriod=25
Nov 03 20:04:45.503684 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.503645    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-home\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-home\") pod \"79a95827-f3b5-4c1d-8c1f-419924011fe6\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") "
Nov 03 20:04:45.503841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.503749    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-tmp-dir\") pod \"79a95827-f3b5-4c1d-8c1f-419924011fe6\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") "
Nov 03 20:04:45.503841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.503786    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-dshm\") pod \"79a95827-f3b5-4c1d-8c1f-419924011fe6\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") "
Nov 03 20:04:45.503841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.503830    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a95827-f3b5-4c1d-8c1f-419924011fe6-kube-api-access-vpphq\" (UniqueName: \"kubernetes.io/projected/79a95827-f3b5-4c1d-8c1f-419924011fe6-kube-api-access-vpphq\") pod \"79a95827-f3b5-4c1d-8c1f-419924011fe6\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") "
Nov 03 20:04:45.503993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.503935    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79a95827-f3b5-4c1d-8c1f-419924011fe6-tls-certs\" (UniqueName: \"kubernetes.io/secret/79a95827-f3b5-4c1d-8c1f-419924011fe6-tls-certs\") pod \"79a95827-f3b5-4c1d-8c1f-419924011fe6\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") "
Nov 03 20:04:45.503993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.503949    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-home" pod "79a95827-f3b5-4c1d-8c1f-419924011fe6" (UID: "79a95827-f3b5-4c1d-8c1f-419924011fe6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:45.503993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.503987    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-model-cache\") pod \"79a95827-f3b5-4c1d-8c1f-419924011fe6\" (UID: \"79a95827-f3b5-4c1d-8c1f-419924011fe6\") "
Nov 03 20:04:45.504158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.503989    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-tmp-dir" pod "79a95827-f3b5-4c1d-8c1f-419924011fe6" (UID: "79a95827-f3b5-4c1d-8c1f-419924011fe6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:45.504327 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.504303    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-model-cache" pod "79a95827-f3b5-4c1d-8c1f-419924011fe6" (UID: "79a95827-f3b5-4c1d-8c1f-419924011fe6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:45.504745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.504726    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:45.504745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.504746    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:45.504916 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.504756    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:45.506227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.506203    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-dshm" pod "79a95827-f3b5-4c1d-8c1f-419924011fe6" (UID: "79a95827-f3b5-4c1d-8c1f-419924011fe6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:45.506227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.506212    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a95827-f3b5-4c1d-8c1f-419924011fe6-kube-api-access-vpphq" pod "79a95827-f3b5-4c1d-8c1f-419924011fe6" (UID: "79a95827-f3b5-4c1d-8c1f-419924011fe6"). InnerVolumeSpecName "kube-api-access-vpphq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:45.506358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.506248    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79a95827-f3b5-4c1d-8c1f-419924011fe6-tls-certs" pod "79a95827-f3b5-4c1d-8c1f-419924011fe6" (UID: "79a95827-f3b5-4c1d-8c1f-419924011fe6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:45.606591 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.606558    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79a95827-f3b5-4c1d-8c1f-419924011fe6-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:45.606591 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.606593    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpphq\" (UniqueName: \"kubernetes.io/projected/79a95827-f3b5-4c1d-8c1f-419924011fe6-kube-api-access-vpphq\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:45.606800 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.606611    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79a95827-f3b5-4c1d-8c1f-419924011fe6-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:45.749193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.749153    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 20:04:45.803384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.803354    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 20:04:45.809801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.809773    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01880166-1848-4cae-a23b-8ffc842ed06c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01880166-1848-4cae-a23b-8ffc842ed06c-plugins-config-volume\") pod \"01880166-1848-4cae-a23b-8ffc842ed06c\" (UID: \"01880166-1848-4cae-a23b-8ffc842ed06c\") "
Nov 03 20:04:45.809944 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.809890    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01880166-1848-4cae-a23b-8ffc842ed06c-kube-api-access-ksq5m\" (UniqueName: \"kubernetes.io/projected/01880166-1848-4cae-a23b-8ffc842ed06c-kube-api-access-ksq5m\") pod \"01880166-1848-4cae-a23b-8ffc842ed06c\" (UID: \"01880166-1848-4cae-a23b-8ffc842ed06c\") "
Nov 03 20:04:45.810272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.810243    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01880166-1848-4cae-a23b-8ffc842ed06c-plugins-config-volume" pod "01880166-1848-4cae-a23b-8ffc842ed06c" (UID: "01880166-1848-4cae-a23b-8ffc842ed06c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:45.810761 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.810729    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01880166-1848-4cae-a23b-8ffc842ed06c-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:45.812745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.812718    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01880166-1848-4cae-a23b-8ffc842ed06c-kube-api-access-ksq5m" pod "01880166-1848-4cae-a23b-8ffc842ed06c" (UID: "01880166-1848-4cae-a23b-8ffc842ed06c"). InnerVolumeSpecName "kube-api-access-ksq5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:45.832361 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.832327    4531 generic.go:356] "Generic (PLEG): container finished" podID="01880166-1848-4cae-a23b-8ffc842ed06c" containerID="ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b" exitCode=0
Nov 03 20:04:45.832527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.832372    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"
Nov 03 20:04:45.832527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.832416    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq" event={"ID":"01880166-1848-4cae-a23b-8ffc842ed06c","Type":"ContainerDied","Data":"ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b"}
Nov 03 20:04:45.832527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.832480    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq" event={"ID":"01880166-1848-4cae-a23b-8ffc842ed06c","Type":"ContainerDied","Data":"f221b0ffb33e88dc1ea2885d5c71e12e3b1f3252ef603153a193cec13944b82a"}
Nov 03 20:04:45.832527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.832509    4531 scope.go:122] "RemoveContainer" containerID="ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b"
Nov 03 20:04:45.833969 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.833943    4531 generic.go:356] "Generic (PLEG): container finished" podID="79a95827-f3b5-4c1d-8c1f-419924011fe6" containerID="4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084" exitCode=0
Nov 03 20:04:45.834103 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.833991    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"
Nov 03 20:04:45.834103 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.834034    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc" event={"ID":"79a95827-f3b5-4c1d-8c1f-419924011fe6","Type":"ContainerDied","Data":"4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084"}
Nov 03 20:04:45.834263 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.834105    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc" event={"ID":"79a95827-f3b5-4c1d-8c1f-419924011fe6","Type":"ContainerDied","Data":"21fabbb413080afee5e10869db6cac048bd4b3bfab5b84f566639f0e48bf183e"}
Nov 03 20:04:45.846236 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.842021    4531 generic.go:356] "Generic (PLEG): container finished" podID="73377b59-5b9c-4cb9-8c69-413b5c215d30" containerID="ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f" exitCode=0
Nov 03 20:04:45.846236 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.842195    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r" event={"ID":"73377b59-5b9c-4cb9-8c69-413b5c215d30","Type":"ContainerDied","Data":"ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f"}
Nov 03 20:04:45.846236 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.842245    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r" event={"ID":"73377b59-5b9c-4cb9-8c69-413b5c215d30","Type":"ContainerDied","Data":"acd069bcbad9c5a6eb803e9147eb479beea95686aecc5d302e1c44f11956d2dc"}
Nov 03 20:04:45.846951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.846735    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"
Nov 03 20:04:45.849288 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.849250    4531 scope.go:122] "RemoveContainer" containerID="ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b"
Nov 03 20:04:45.849670 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:45.849639    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b\": container with ID starting with ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b not found: ID does not exist" containerID="ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b"
Nov 03 20:04:45.849758 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.849710    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b"} err="failed to get container status \"ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b\": rpc error: code = NotFound desc = could not find container \"ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b\": container with ID starting with ab1bfe904fffcbdd21b11cec78bfb39c1918b5365d2aa59b5606dc4c4004d38b not found: ID does not exist"
Nov 03 20:04:45.849758 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.849737    4531 scope.go:122] "RemoveContainer" containerID="4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084"
Nov 03 20:04:45.867265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.867142    4531 scope.go:122] "RemoveContainer" containerID="4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084"
Nov 03 20:04:45.867590 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:45.867562    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084\": container with ID starting with 4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084 not found: ID does not exist" containerID="4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084"
Nov 03 20:04:45.867682 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.867603    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084"} err="failed to get container status \"4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084\": rpc error: code = NotFound desc = could not find container \"4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084\": container with ID starting with 4c340f50ca5e63a8a392e2e4192eacade24baa501d77d096ef6499512ef65084 not found: ID does not exist"
Nov 03 20:04:45.867682 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.867626    4531 scope.go:122] "RemoveContainer" containerID="ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f"
Nov 03 20:04:45.876290 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.876268    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"]
Nov 03 20:04:45.884311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.884289    4531 scope.go:122] "RemoveContainer" containerID="ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f"
Nov 03 20:04:45.902433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.886252    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-32a584/auth-test-model-32a584-kserve-bc8cc974f-8cmlc"]
Nov 03 20:04:45.947199 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:45.930041    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f\": container with ID starting with ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f not found: ID does not exist" containerID="ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f"
Nov 03 20:04:45.947199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.930124    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f"} err="failed to get container status \"ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f\": rpc error: code = NotFound desc = could not find container \"ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f\": container with ID starting with ff0ad09f788583538130d92ebc6c2179c255e8e31706f6e8262b6ac0b7d8af4f not found: ID does not exist"
Nov 03 20:04:45.947199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.930669    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:45.947199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.932084    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73377b59-5b9c-4cb9-8c69-413b5c215d30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73377b59-5b9c-4cb9-8c69-413b5c215d30-plugins-config-volume\") pod \"73377b59-5b9c-4cb9-8c69-413b5c215d30\" (UID: \"73377b59-5b9c-4cb9-8c69-413b5c215d30\") "
Nov 03 20:04:45.947199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.932545    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73377b59-5b9c-4cb9-8c69-413b5c215d30-plugins-config-volume" pod "73377b59-5b9c-4cb9-8c69-413b5c215d30" (UID: "73377b59-5b9c-4cb9-8c69-413b5c215d30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:45.947199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.932956    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73377b59-5b9c-4cb9-8c69-413b5c215d30-kube-api-access-q5vgn\" (UniqueName: \"kubernetes.io/projected/73377b59-5b9c-4cb9-8c69-413b5c215d30-kube-api-access-q5vgn\") pod \"73377b59-5b9c-4cb9-8c69-413b5c215d30\" (UID: \"73377b59-5b9c-4cb9-8c69-413b5c215d30\") "
Nov 03 20:04:45.947199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.934016    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73377b59-5b9c-4cb9-8c69-413b5c215d30-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:45.947199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.934040    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksq5m\" (UniqueName: \"kubernetes.io/projected/01880166-1848-4cae-a23b-8ffc842ed06c-kube-api-access-ksq5m\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:45.960730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.950416    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"]
Nov 03 20:04:45.960730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.950727    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73377b59-5b9c-4cb9-8c69-413b5c215d30-kube-api-access-q5vgn" pod "73377b59-5b9c-4cb9-8c69-413b5c215d30" (UID: "73377b59-5b9c-4cb9-8c69-413b5c215d30"). InnerVolumeSpecName "kube-api-access-q5vgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:45.960730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:45.955971    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f3188-6575d78977-4wsvq"]
Nov 03 20:04:46.036036 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:46.036005    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5vgn\" (UniqueName: \"kubernetes.io/projected/73377b59-5b9c-4cb9-8c69-413b5c215d30-kube-api-access-q5vgn\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:46.195323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:46.195286    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"]
Nov 03 20:04:46.202566 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:46.202539    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f3188-5f8dd694c-jzm2r"]
Nov 03 20:04:46.580683 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:46.580591    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:46.721830 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:46.721795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:46.829750 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:46.829705    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"]
Nov 03 20:04:46.830097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:46.829950    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy" containerID="cri-o://07339454dd278a19d4fc64a22b8d56760f847d90a49e8eb4a3bd13531ac2243f" gracePeriod=30
Nov 03 20:04:46.864231 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:46.864140    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01880166-1848-4cae-a23b-8ffc842ed06c" path="/var/lib/kubelet/pods/01880166-1848-4cae-a23b-8ffc842ed06c/volumes"
Nov 03 20:04:46.864645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:46.864629    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73377b59-5b9c-4cb9-8c69-413b5c215d30" path="/var/lib/kubelet/pods/73377b59-5b9c-4cb9-8c69-413b5c215d30/volumes"
Nov 03 20:04:46.865280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:46.865262    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a95827-f3b5-4c1d-8c1f-419924011fe6" path="/var/lib/kubelet/pods/79a95827-f3b5-4c1d-8c1f-419924011fe6/volumes"
Nov 03 20:04:46.911239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:46.911206    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:47.313908 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.313231    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"]
Nov 03 20:04:47.313908 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.313482    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv" podUID="062411ad-e3e7-4e1d-b7b4-c5dba546e973" containerName="maas-api" containerID="cri-o://5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b" gracePeriod=30
Nov 03 20:04:47.412671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.412639    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"]
Nov 03 20:04:47.414057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.414015    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:47.419412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419376    4531 cpu_manager.go:414] "containerMap: removing container" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerName="istio-proxy"
Nov 03 20:04:47.419412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419412    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerName="istio-proxy"
Nov 03 20:04:47.419614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419466    4531 cpu_manager.go:414] "containerMap: removing container" podUID="90e841f9-77b1-4f9e-bf90-a444d25f01d7" containerName="maas-api"
Nov 03 20:04:47.419614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419474    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e841f9-77b1-4f9e-bf90-a444d25f01d7" containerName="maas-api"
Nov 03 20:04:47.419614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419501    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy"
Nov 03 20:04:47.419614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419514    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy"
Nov 03 20:04:47.419614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419550    4531 cpu_manager.go:414] "containerMap: removing container" podUID="01880166-1848-4cae-a23b-8ffc842ed06c" containerName="payload-processing"
Nov 03 20:04:47.419614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419558    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01880166-1848-4cae-a23b-8ffc842ed06c" containerName="payload-processing"
Nov 03 20:04:47.419614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419595    4531 cpu_manager.go:414] "containerMap: removing container" podUID="73377b59-5b9c-4cb9-8c69-413b5c215d30" containerName="payload-pre-processing"
Nov 03 20:04:47.419614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419603    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73377b59-5b9c-4cb9-8c69-413b5c215d30" containerName="payload-pre-processing"
Nov 03 20:04:47.419994 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419656    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" containerName="istio-proxy"
Nov 03 20:04:47.419994 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419664    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" containerName="istio-proxy"
Nov 03 20:04:47.419994 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419698    4531 cpu_manager.go:414] "containerMap: removing container" podUID="79a95827-f3b5-4c1d-8c1f-419924011fe6" containerName="main"
Nov 03 20:04:47.419994 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419706    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a95827-f3b5-4c1d-8c1f-419924011fe6" containerName="main"
Nov 03 20:04:47.419994 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419759    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8fa5ca17-a405-43b0-bf42-acdf79ff1a70" containerName="main"
Nov 03 20:04:47.419994 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.419766    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa5ca17-a405-43b0-bf42-acdf79ff1a70" containerName="main"
Nov 03 20:04:47.420833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.420134    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2400a16-c022-45cb-91ac-8fb743616e13" containerName="istio-proxy"
Nov 03 20:04:47.420833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.420189    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="01880166-1848-4cae-a23b-8ffc842ed06c" containerName="payload-processing"
Nov 03 20:04:47.420833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.420210    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="73377b59-5b9c-4cb9-8c69-413b5c215d30" containerName="payload-pre-processing"
Nov 03 20:04:47.420833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.420234    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6364721-945b-4bfc-b118-cf93b16619d3" containerName="istio-proxy"
Nov 03 20:04:47.420833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.420254    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a95827-f3b5-4c1d-8c1f-419924011fe6" containerName="main"
Nov 03 20:04:47.420833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.420267    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa5ca17-a405-43b0-bf42-acdf79ff1a70" containerName="main"
Nov 03 20:04:47.420833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.420284    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="595c6bfd-72b7-4501-95ce-5c7e3eda018c" containerName="istio-proxy"
Nov 03 20:04:47.420833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.420296    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e841f9-77b1-4f9e-bf90-a444d25f01d7" containerName="maas-api"
Nov 03 20:04:47.426914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.426868    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"]
Nov 03 20:04:47.427257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.427236    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:47.446390 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.446360    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:04:47.446390 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.446376    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:04:47.574100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.573994    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:47.574100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.574066    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:47.574377 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.574115    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqp7f\" (UniqueName: \"kubernetes.io/projected/220d5899-b75d-4842-b3d4-e7caa9d251c5-kube-api-access-zqp7f\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:47.644282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.644248    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 20:04:47.650585 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.650543    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:47.676460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.676421    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:47.676725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.676488    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:47.676725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.676537    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqp7f\" (UniqueName: \"kubernetes.io/projected/220d5899-b75d-4842-b3d4-e7caa9d251c5-kube-api-access-zqp7f\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:47.677307 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.677280    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:47.679016 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.678988    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:47.684224 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.684196    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqp7f\" (UniqueName: \"kubernetes.io/projected/220d5899-b75d-4842-b3d4-e7caa9d251c5-kube-api-access-zqp7f\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:47.769405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.769345    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:47.778288 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.778262    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-maas-api-tls\") pod \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") "
Nov 03 20:04:47.778411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.778354    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/062411ad-e3e7-4e1d-b7b4-c5dba546e973-kube-api-access-4wsqz\" (UniqueName: \"kubernetes.io/projected/062411ad-e3e7-4e1d-b7b4-c5dba546e973-kube-api-access-4wsqz\") pod \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") "
Nov 03 20:04:47.778411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.778404    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-metrics-certs\" (UniqueName: \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-metrics-certs\") pod \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") "
Nov 03 20:04:47.778537 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.778514    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/062411ad-e3e7-4e1d-b7b4-c5dba546e973-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/062411ad-e3e7-4e1d-b7b4-c5dba546e973-openshift-service-ca\") pod \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\" (UID: \"062411ad-e3e7-4e1d-b7b4-c5dba546e973\") "
Nov 03 20:04:47.778941 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.778912    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062411ad-e3e7-4e1d-b7b4-c5dba546e973-openshift-service-ca" pod "062411ad-e3e7-4e1d-b7b4-c5dba546e973" (UID: "062411ad-e3e7-4e1d-b7b4-c5dba546e973"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:47.779220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.779201    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/062411ad-e3e7-4e1d-b7b4-c5dba546e973-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:47.781029 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.781001    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-maas-api-tls" pod "062411ad-e3e7-4e1d-b7b4-c5dba546e973" (UID: "062411ad-e3e7-4e1d-b7b4-c5dba546e973"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:47.781126 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.781032    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062411ad-e3e7-4e1d-b7b4-c5dba546e973-kube-api-access-4wsqz" pod "062411ad-e3e7-4e1d-b7b4-c5dba546e973" (UID: "062411ad-e3e7-4e1d-b7b4-c5dba546e973"). InnerVolumeSpecName "kube-api-access-4wsqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:47.781126 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.781085    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-metrics-certs" pod "062411ad-e3e7-4e1d-b7b4-c5dba546e973" (UID: "062411ad-e3e7-4e1d-b7b4-c5dba546e973"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:47.788884 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.788806    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:47.792453 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.792429    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"]
Nov 03 20:04:47.796867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:47.796846    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f58171-h5n4h"]
Nov 03 20:04:48.010193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.008959    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:48.010193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.008998    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/062411ad-e3e7-4e1d-b7b4-c5dba546e973-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:48.010193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.009016    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wsqz\" (UniqueName: \"kubernetes.io/projected/062411ad-e3e7-4e1d-b7b4-c5dba546e973-kube-api-access-4wsqz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:48.129218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.115926    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:48.157455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.144161    4531 generic.go:356] "Generic (PLEG): container finished" podID="062411ad-e3e7-4e1d-b7b4-c5dba546e973" containerID="5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b" exitCode=0
Nov 03 20:04:48.157455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.144461    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv" event={"ID":"062411ad-e3e7-4e1d-b7b4-c5dba546e973","Type":"ContainerDied","Data":"5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b"}
Nov 03 20:04:48.157455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.144811    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv" event={"ID":"062411ad-e3e7-4e1d-b7b4-c5dba546e973","Type":"ContainerDied","Data":"3ca2f14cf08263d63ddc4e35d93e6526f2cf29a210c55318adca6409519d6c38"}
Nov 03 20:04:48.157455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.144863    4531 scope.go:122] "RemoveContainer" containerID="5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b"
Nov 03 20:04:48.157455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.145025    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"
Nov 03 20:04:48.170211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.170157    4531 scope.go:122] "RemoveContainer" containerID="5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b"
Nov 03 20:04:48.170558 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:48.170531    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b\": container with ID starting with 5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b not found: ID does not exist" containerID="5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b"
Nov 03 20:04:48.170653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.170567    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b"} err="failed to get container status \"5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b\": rpc error: code = NotFound desc = could not find container \"5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b\": container with ID starting with 5afba1851c3518d8799b6b4954104c433d0b24e1fc6f80b238b0f2330f090b7b not found: ID does not exist"
Nov 03 20:04:48.180067 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.180035    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"]
Nov 03 20:04:48.188960 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.188919    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f58171-5c5f799f57-lzbpv"]
Nov 03 20:04:48.276807 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.276770    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:48.297225 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:04:48.297168    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220d5899_b75d_4842_b3d4_e7caa9d251c5.slice/crio-42dc928cd0c96db55e32271452654f3b43636039a1b05edc16499c5c8ef0bb0d WatchSource:0}: Error finding container 42dc928cd0c96db55e32271452654f3b43636039a1b05edc16499c5c8ef0bb0d: Status 404 returned error can't find the container with id 42dc928cd0c96db55e32271452654f3b43636039a1b05edc16499c5c8ef0bb0d
Nov 03 20:04:48.297352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.297271    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"]
Nov 03 20:04:48.427014 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.426933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:48.861389 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.861357    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="062411ad-e3e7-4e1d-b7b4-c5dba546e973" path="/var/lib/kubelet/pods/062411ad-e3e7-4e1d-b7b4-c5dba546e973/volumes"
Nov 03 20:04:48.861894 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.861877    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c38b4b32-93b2-4b98-9233-02442a28118d" path="/var/lib/kubelet/pods/c38b4b32-93b2-4b98-9233-02442a28118d/volumes"
Nov 03 20:04:48.972765 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:48.972677    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:49.107706 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:49.107676    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:49.158145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:49.158103    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx" event={"ID":"220d5899-b75d-4842-b3d4-e7caa9d251c5","Type":"ContainerStarted","Data":"0606ebb42f091d77a9e1a68cce0f56d6a6f2389863e7c90585ea07ebb799ca5f"}
Nov 03 20:04:49.158354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:49.158223    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx" event={"ID":"220d5899-b75d-4842-b3d4-e7caa9d251c5","Type":"ContainerStarted","Data":"42dc928cd0c96db55e32271452654f3b43636039a1b05edc16499c5c8ef0bb0d"}
Nov 03 20:04:49.176499 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:49.176449    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx" podStartSLOduration=2.176427459 podStartE2EDuration="2.176427459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:49.17199183 +0000 UTC m=+2002.442115126" watchObservedRunningTime="2026-11-03 20:04:49.176427459 +0000 UTC m=+2002.446550753"
Nov 03 20:04:49.219118 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:49.219084    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"]
Nov 03 20:04:49.219379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:49.219353    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerName="istio-proxy" containerID="cri-o://56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb" gracePeriod=30
Nov 03 20:04:49.288438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:49.288412    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:50.057980 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:50.057949    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:50.169716 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:50.169682    4531 generic.go:356] "Generic (PLEG): container finished" podID="220d5899-b75d-4842-b3d4-e7caa9d251c5" containerID="0606ebb42f091d77a9e1a68cce0f56d6a6f2389863e7c90585ea07ebb799ca5f" exitCode=0
Nov 03 20:04:50.169910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:50.169755    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx" event={"ID":"220d5899-b75d-4842-b3d4-e7caa9d251c5","Type":"ContainerDied","Data":"0606ebb42f091d77a9e1a68cce0f56d6a6f2389863e7c90585ea07ebb799ca5f"}
Nov 03 20:04:50.636583 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:50.636543    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:04:51.078601 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.077338    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:51.350682 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.350620    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:51.426024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.425987    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220d5899-b75d-4842-b3d4-e7caa9d251c5-kube-api-access-zqp7f\" (UniqueName: \"kubernetes.io/projected/220d5899-b75d-4842-b3d4-e7caa9d251c5-kube-api-access-zqp7f\") pod \"220d5899-b75d-4842-b3d4-e7caa9d251c5\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") "
Nov 03 20:04:51.426286 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.426137    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-xks-ca\") pod \"220d5899-b75d-4842-b3d4-e7caa9d251c5\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") "
Nov 03 20:04:51.426369 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.426304    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-openshift-ca\") pod \"220d5899-b75d-4842-b3d4-e7caa9d251c5\" (UID: \"220d5899-b75d-4842-b3d4-e7caa9d251c5\") "
Nov 03 20:04:51.426902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.426871    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-openshift-ca" pod "220d5899-b75d-4842-b3d4-e7caa9d251c5" (UID: "220d5899-b75d-4842-b3d4-e7caa9d251c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:51.427249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.427222    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:51.428648 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.428622    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220d5899-b75d-4842-b3d4-e7caa9d251c5-kube-api-access-zqp7f" pod "220d5899-b75d-4842-b3d4-e7caa9d251c5" (UID: "220d5899-b75d-4842-b3d4-e7caa9d251c5"). InnerVolumeSpecName "kube-api-access-zqp7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:51.428864 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.428838    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-xks-ca" pod "220d5899-b75d-4842-b3d4-e7caa9d251c5" (UID: "220d5899-b75d-4842-b3d4-e7caa9d251c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:51.528803 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.528766    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqp7f\" (UniqueName: \"kubernetes.io/projected/220d5899-b75d-4842-b3d4-e7caa9d251c5-kube-api-access-zqp7f\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:51.528803 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.528806    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220d5899-b75d-4842-b3d4-e7caa9d251c5-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:51.881780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:51.881737    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:52.194603 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.193379    4531 generic.go:356] "Generic (PLEG): container finished" podID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerID="07339454dd278a19d4fc64a22b8d56760f847d90a49e8eb4a3bd13531ac2243f" exitCode=0
Nov 03 20:04:52.194603 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.193461    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" event={"ID":"96b9db67-2b47-454f-8e19-d892508e1b7c","Type":"ContainerDied","Data":"07339454dd278a19d4fc64a22b8d56760f847d90a49e8eb4a3bd13531ac2243f"}
Nov 03 20:04:52.194603 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.193946    4531 scope.go:122] "RemoveContainer" containerID="f07f96380fc76cc384ce45bda54665d97d04862c90f568d2747dd310d7268057"
Nov 03 20:04:52.195515 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.195489    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx" event={"ID":"220d5899-b75d-4842-b3d4-e7caa9d251c5","Type":"ContainerDied","Data":"42dc928cd0c96db55e32271452654f3b43636039a1b05edc16499c5c8ef0bb0d"}
Nov 03 20:04:52.195628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.195534    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42dc928cd0c96db55e32271452654f3b43636039a1b05edc16499c5c8ef0bb0d"
Nov 03 20:04:52.195628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.195590    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"
Nov 03 20:04:52.306159 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.306129    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 20:04:52.461088 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.461049    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-certs\") pod \"96b9db67-2b47-454f-8e19-d892508e1b7c\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") "
Nov 03 20:04:52.461322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.461279    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-envoy\") pod \"96b9db67-2b47-454f-8e19-d892508e1b7c\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") "
Nov 03 20:04:52.461400 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.461350    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-socket\") pod \"96b9db67-2b47-454f-8e19-d892508e1b7c\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") "
Nov 03 20:04:52.461400 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.461368    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-certs" pod "96b9db67-2b47-454f-8e19-d892508e1b7c" (UID: "96b9db67-2b47-454f-8e19-d892508e1b7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:52.461400 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.461389    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-podinfo\") pod \"96b9db67-2b47-454f-8e19-d892508e1b7c\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") "
Nov 03 20:04:52.461560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.461503    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-token\" (UniqueName: \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-token\") pod \"96b9db67-2b47-454f-8e19-d892508e1b7c\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") "
Nov 03 20:04:52.461560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.461546    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-socket" pod "96b9db67-2b47-454f-8e19-d892508e1b7c" (UID: "96b9db67-2b47-454f-8e19-d892508e1b7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:52.461670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.461575    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b9db67-2b47-454f-8e19-d892508e1b7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b9db67-2b47-454f-8e19-d892508e1b7c-istiod-ca-cert\") pod \"96b9db67-2b47-454f-8e19-d892508e1b7c\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") "
Nov 03 20:04:52.461728 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.461675    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-credential-socket\") pod \"96b9db67-2b47-454f-8e19-d892508e1b7c\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") "
Nov 03 20:04:52.461786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.461725    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-data\") pod \"96b9db67-2b47-454f-8e19-d892508e1b7c\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") "
Nov 03 20:04:52.461786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.461763    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-kube-api-access-hr5fk\" (UniqueName: \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-kube-api-access-hr5fk\") pod \"96b9db67-2b47-454f-8e19-d892508e1b7c\" (UID: \"96b9db67-2b47-454f-8e19-d892508e1b7c\") "
Nov 03 20:04:52.462088 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.462052    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b9db67-2b47-454f-8e19-d892508e1b7c-istiod-ca-cert" pod "96b9db67-2b47-454f-8e19-d892508e1b7c" (UID: "96b9db67-2b47-454f-8e19-d892508e1b7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:52.462613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.462585    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:52.462718 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.462618    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b9db67-2b47-454f-8e19-d892508e1b7c-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:52.462718 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.462635    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:52.462718 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.462702    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-credential-socket" pod "96b9db67-2b47-454f-8e19-d892508e1b7c" (UID: "96b9db67-2b47-454f-8e19-d892508e1b7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:52.463427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.463398    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-data" pod "96b9db67-2b47-454f-8e19-d892508e1b7c" (UID: "96b9db67-2b47-454f-8e19-d892508e1b7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:52.464655 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.464618    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-podinfo" pod "96b9db67-2b47-454f-8e19-d892508e1b7c" (UID: "96b9db67-2b47-454f-8e19-d892508e1b7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:04:52.464755 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.464690    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-envoy" pod "96b9db67-2b47-454f-8e19-d892508e1b7c" (UID: "96b9db67-2b47-454f-8e19-d892508e1b7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:52.464898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.464876    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-kube-api-access-hr5fk" pod "96b9db67-2b47-454f-8e19-d892508e1b7c" (UID: "96b9db67-2b47-454f-8e19-d892508e1b7c"). InnerVolumeSpecName "kube-api-access-hr5fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:52.465279 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.465255    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-token" pod "96b9db67-2b47-454f-8e19-d892508e1b7c" (UID: "96b9db67-2b47-454f-8e19-d892508e1b7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:52.563812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.563778    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:52.563812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.563808    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:52.563812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.563818    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:52.564056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.563827    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:52.564056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.563835    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b9db67-2b47-454f-8e19-d892508e1b7c-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:52.564056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.563846    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr5fk\" (UniqueName: \"kubernetes.io/projected/96b9db67-2b47-454f-8e19-d892508e1b7c-kube-api-access-hr5fk\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:52.947029 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:52.946166    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:53.208541 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.208432    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4" event={"ID":"96b9db67-2b47-454f-8e19-d892508e1b7c","Type":"ContainerDied","Data":"d54dfae83bd5503118898c036d565a25e5b0c62a17d44755015348726b724b71"}
Nov 03 20:04:53.208541 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.208517    4531 scope.go:122] "RemoveContainer" containerID="07339454dd278a19d4fc64a22b8d56760f847d90a49e8eb4a3bd13531ac2243f"
Nov 03 20:04:53.209105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.208654    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"
Nov 03 20:04:53.227279 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.227254    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"]
Nov 03 20:04:53.251514 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.234009    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-x9ck4"]
Nov 03 20:04:53.540246 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.539579    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"]
Nov 03 20:04:53.624617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.624586    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:53.721790 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.721754    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"]
Nov 03 20:04:53.722969 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.722944    4531 cpu_manager.go:414] "containerMap: removing container" podUID="062411ad-e3e7-4e1d-b7b4-c5dba546e973" containerName="maas-api"
Nov 03 20:04:53.723065 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.722972    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062411ad-e3e7-4e1d-b7b4-c5dba546e973" containerName="maas-api"
Nov 03 20:04:53.723065 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.723038    4531 cpu_manager.go:414] "containerMap: removing container" podUID="220d5899-b75d-4842-b3d4-e7caa9d251c5" containerName="revoke-keys"
Nov 03 20:04:53.723065 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.723046    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220d5899-b75d-4842-b3d4-e7caa9d251c5" containerName="revoke-keys"
Nov 03 20:04:53.723214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.723065    4531 cpu_manager.go:414] "containerMap: removing container" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy"
Nov 03 20:04:53.723214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.723073    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy"
Nov 03 20:04:53.723214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.723100    4531 cpu_manager.go:414] "containerMap: removing container" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy"
Nov 03 20:04:53.723214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.723108    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy"
Nov 03 20:04:53.723362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.723302    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="220d5899-b75d-4842-b3d4-e7caa9d251c5" containerName="revoke-keys"
Nov 03 20:04:53.723362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.723315    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy"
Nov 03 20:04:53.723362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.723329    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" containerName="istio-proxy"
Nov 03 20:04:53.723362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.723344    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="062411ad-e3e7-4e1d-b7b4-c5dba546e973" containerName="maas-api"
Nov 03 20:04:53.731239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.731211    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:53.735391 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.734236    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:04:53.735391 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.734371    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:04:53.742485 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.742456    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"]
Nov 03 20:04:53.812527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.812430    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:53.812683 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.812568    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:53.812683 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.812629    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k584q\" (UniqueName: \"kubernetes.io/projected/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-kube-api-access-k584q\") pod \"maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:53.831443 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.831402    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:53.920196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.915270    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:53.920196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.915360    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k584q\" (UniqueName: \"kubernetes.io/projected/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-kube-api-access-k584q\") pod \"maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:53.920196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.915784    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:53.934210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:53.932374    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:54.025214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.024635    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:54.054193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.053889    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k584q\" (UniqueName: \"kubernetes.io/projected/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-kube-api-access-k584q\") pod \"maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:54.082597 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.082496    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:54.435740 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.435694    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"]
Nov 03 20:04:54.438166 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:04:54.438137    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcebb6a_d39d_4eba_ba2c_5dab051b4dad.slice/crio-83b2341622ef24bbd1204822b7e5a01ee35f5395a30227a1d815798fadebe784 WatchSource:0}: Error finding container 83b2341622ef24bbd1204822b7e5a01ee35f5395a30227a1d815798fadebe784: Status 404 returned error can't find the container with id 83b2341622ef24bbd1204822b7e5a01ee35f5395a30227a1d815798fadebe784
Nov 03 20:04:54.491405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.491380    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:54.533999 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.533961    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istiod-ca-cert\") pod \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") "
Nov 03 20:04:54.534231 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.534153    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-podinfo\") pod \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") "
Nov 03 20:04:54.534231 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.534218    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-data\") pod \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") "
Nov 03 20:04:54.534384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.534280    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-credential-socket\") pod \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") "
Nov 03 20:04:54.534444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.534405    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-envoy\") pod \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") "
Nov 03 20:04:54.534444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.534416    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istiod-ca-cert" pod "9edf055d-9f1e-415e-a672-8cfd8e4d2263" (UID: "9edf055d-9f1e-415e-a672-8cfd8e4d2263"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:54.534564 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.534495    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-socket\") pod \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") "
Nov 03 20:04:54.534632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.534571    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-certs\") pod \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") "
Nov 03 20:04:54.534632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.534601    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-kube-api-access-hrgns\" (UniqueName: \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-kube-api-access-hrgns\") pod \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") "
Nov 03 20:04:54.534749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.534651    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-token\" (UniqueName: \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-token\") pod \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\" (UID: \"9edf055d-9f1e-415e-a672-8cfd8e4d2263\") "
Nov 03 20:04:54.534749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.534677    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-data" pod "9edf055d-9f1e-415e-a672-8cfd8e4d2263" (UID: "9edf055d-9f1e-415e-a672-8cfd8e4d2263"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:54.534918 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.534894    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-socket" pod "9edf055d-9f1e-415e-a672-8cfd8e4d2263" (UID: "9edf055d-9f1e-415e-a672-8cfd8e4d2263"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:54.535097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.535077    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-credential-socket" pod "9edf055d-9f1e-415e-a672-8cfd8e4d2263" (UID: "9edf055d-9f1e-415e-a672-8cfd8e4d2263"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:54.535506 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.535329    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-certs" pod "9edf055d-9f1e-415e-a672-8cfd8e4d2263" (UID: "9edf055d-9f1e-415e-a672-8cfd8e4d2263"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:54.535620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.535532    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.535620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.535556    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.535620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.535571    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.535620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.535600    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.535620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.535614    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.536758 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.536734    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-podinfo" pod "9edf055d-9f1e-415e-a672-8cfd8e4d2263" (UID: "9edf055d-9f1e-415e-a672-8cfd8e4d2263"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:04:54.538459 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.538370    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-token" pod "9edf055d-9f1e-415e-a672-8cfd8e4d2263" (UID: "9edf055d-9f1e-415e-a672-8cfd8e4d2263"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:54.538563 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.538459    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-envoy" pod "9edf055d-9f1e-415e-a672-8cfd8e4d2263" (UID: "9edf055d-9f1e-415e-a672-8cfd8e4d2263"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:54.539351 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.539325    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-kube-api-access-hrgns" pod "9edf055d-9f1e-415e-a672-8cfd8e4d2263" (UID: "9edf055d-9f1e-415e-a672-8cfd8e4d2263"). InnerVolumeSpecName "kube-api-access-hrgns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:54.637995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.637956    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.637995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.637993    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.638296 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.638006    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrgns\" (UniqueName: \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-kube-api-access-hrgns\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.638296 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.638017    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edf055d-9f1e-415e-a672-8cfd8e4d2263-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.861994 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.861966    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b9db67-2b47-454f-8e19-d892508e1b7c" path="/var/lib/kubelet/pods/96b9db67-2b47-454f-8e19-d892508e1b7c/volumes"
Nov 03 20:04:54.930352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:54.924842    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:55.176935 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.176898    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:55.251585 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.251496    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n" event={"ID":"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad","Type":"ContainerStarted","Data":"d6bc67d8a54c11077373295037b210c075726e69f64211ceb34708e014336d0e"}
Nov 03 20:04:55.251585 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.251549    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n" event={"ID":"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad","Type":"ContainerStarted","Data":"83b2341622ef24bbd1204822b7e5a01ee35f5395a30227a1d815798fadebe784"}
Nov 03 20:04:55.253270 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.253243    4531 generic.go:356] "Generic (PLEG): container finished" podID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerID="56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb" exitCode=0
Nov 03 20:04:55.253404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.253287    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"
Nov 03 20:04:55.253404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.253298    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" event={"ID":"9edf055d-9f1e-415e-a672-8cfd8e4d2263","Type":"ContainerDied","Data":"56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb"}
Nov 03 20:04:55.253404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.253335    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r" event={"ID":"9edf055d-9f1e-415e-a672-8cfd8e4d2263","Type":"ContainerDied","Data":"7dbd35c44d001f180e3a794cd7b92a1d364200b766b9d005c29aaa3d848f6c2f"}
Nov 03 20:04:55.253404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.253357    4531 scope.go:122] "RemoveContainer" containerID="56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb"
Nov 03 20:04:55.264601 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.264579    4531 scope.go:122] "RemoveContainer" containerID="56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb"
Nov 03 20:04:55.264907 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:55.264888    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb\": container with ID starting with 56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb not found: ID does not exist" containerID="56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb"
Nov 03 20:04:55.264955 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.264916    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb"} err="failed to get container status \"56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb\": rpc error: code = NotFound desc = could not find container \"56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb\": container with ID starting with 56ff5308e1443789206e99f7adedf7531d4c090ba5951bae78bd5b281fac3aeb not found: ID does not exist"
Nov 03 20:04:55.273823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.273775    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n" podStartSLOduration=2.273763877 podStartE2EDuration="2.273763877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:55.269633074 +0000 UTC m=+2008.539756368" watchObservedRunningTime="2026-11-03 20:04:55.273763877 +0000 UTC m=+2008.543887171"
Nov 03 20:04:55.286485 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.286459    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"]
Nov 03 20:04:55.290644 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.290622    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-k6c4r"]
Nov 03 20:04:55.308677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.308643    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:55.427989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.425895    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"]
Nov 03 20:04:55.427989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.426139    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" podUID="a24b4233-7e81-47f5-90b3-74601c85022b" containerName="istio-proxy" containerID="cri-o://b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71" gracePeriod=30
Nov 03 20:04:55.500099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.500066    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:55.953210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:55.943467    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:56.265209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:56.265108    4531 generic.go:356] "Generic (PLEG): container finished" podID="2dcebb6a-d39d-4eba-ba2c-5dab051b4dad" containerID="d6bc67d8a54c11077373295037b210c075726e69f64211ceb34708e014336d0e" exitCode=0
Nov 03 20:04:56.265381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:56.265202    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n" event={"ID":"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad","Type":"ContainerDied","Data":"d6bc67d8a54c11077373295037b210c075726e69f64211ceb34708e014336d0e"}
Nov 03 20:04:56.860581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:56.860548    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" path="/var/lib/kubelet/pods/9edf055d-9f1e-415e-a672-8cfd8e4d2263/volumes"
Nov 03 20:04:57.012377 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.012318    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:57.451344 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.451314    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:57.502097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.502050    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-xks-ca\") pod \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") "
Nov 03 20:04:57.502319 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.502196    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-kube-api-access-k584q\" (UniqueName: \"kubernetes.io/projected/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-kube-api-access-k584q\") pod \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") "
Nov 03 20:04:57.502412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.502334    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-openshift-ca\") pod \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\" (UID: \"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad\") "
Nov 03 20:04:57.503358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.503324    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-openshift-ca" pod "2dcebb6a-d39d-4eba-ba2c-5dab051b4dad" (UID: "2dcebb6a-d39d-4eba-ba2c-5dab051b4dad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:57.504798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.504773    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-kube-api-access-k584q" pod "2dcebb6a-d39d-4eba-ba2c-5dab051b4dad" (UID: "2dcebb6a-d39d-4eba-ba2c-5dab051b4dad"). InnerVolumeSpecName "kube-api-access-k584q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:57.505438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.505416    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-xks-ca" pod "2dcebb6a-d39d-4eba-ba2c-5dab051b4dad" (UID: "2dcebb6a-d39d-4eba-ba2c-5dab051b4dad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:57.550964 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.550890    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"]
Nov 03 20:04:57.561217 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.561189    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"]
Nov 03 20:04:57.608831 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.608795    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:57.608831 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.608833    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k584q\" (UniqueName: \"kubernetes.io/projected/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-kube-api-access-k584q\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:57.609128 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.608849    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:57.669932 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.669900    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:57.794460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.794424    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:57.938970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:57.938926    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:58.283926 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:58.283846    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"
Nov 03 20:04:58.283926 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:58.283881    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n" event={"ID":"2dcebb6a-d39d-4eba-ba2c-5dab051b4dad","Type":"ContainerDied","Data":"83b2341622ef24bbd1204822b7e5a01ee35f5395a30227a1d815798fadebe784"}
Nov 03 20:04:58.284116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:58.283933    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83b2341622ef24bbd1204822b7e5a01ee35f5395a30227a1d815798fadebe784"
Nov 03 20:04:58.681283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:58.681235    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:59.427811 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:59.427765    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:59.882653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:59.882623    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"]
Nov 03 20:04:59.897301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:04:59.897269    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"]
Nov 03 20:04:59.996703 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:59.996673    4531 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-w3-32a584: configmap "payload-processing-plugins-e2e-shared-b-w3-32a584" not found
Nov 03 20:04:59.997024 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:04:59.996983    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume podName:6c5f6c12-d596-48e5-90bc-cc0f523a106b nodeName:}" failed. No retries permitted until 2026-11-03 20:05:00.496956961 +0000 UTC m=+2013.767080257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume") pod "payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk" (UID: "6c5f6c12-d596-48e5-90bc-cc0f523a106b") : configmap "payload-processing-plugins-e2e-shared-b-w3-32a584" not found
Nov 03 20:05:00.008544 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:00.008512    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:00.164278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:00.158828    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:00.539606 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:00.539571    4531 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-w3-32a584: configmap "payload-processing-plugins-e2e-shared-b-w3-32a584" not found
Nov 03 20:05:00.540035 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:00.539660    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume podName:6c5f6c12-d596-48e5-90bc-cc0f523a106b nodeName:}" failed. No retries permitted until 2026-11-03 20:05:01.539644182 +0000 UTC m=+2014.809767460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume") pod "payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk" (UID: "6c5f6c12-d596-48e5-90bc-cc0f523a106b") : configmap "payload-processing-plugins-e2e-shared-b-w3-32a584" not found
Nov 03 20:05:00.640085 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:00.640045    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"]
Nov 03 20:05:00.640372 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:00.640333    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr" podUID="211d2b1f-7806-4f29-8c89-35f05f3cd673" containerName="maas-api" containerID="cri-o://d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13" gracePeriod=30
Nov 03 20:05:00.739236 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:00.739199    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:00.789080 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:00.789047    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:05:00.920400 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:00.915546    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:01.108467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.107793    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-credential-socket\") pod \"a24b4233-7e81-47f5-90b3-74601c85022b\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") "
Nov 03 20:05:01.108467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.107953    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-certs\") pod \"a24b4233-7e81-47f5-90b3-74601c85022b\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") "
Nov 03 20:05:01.108467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.107998    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-data\") pod \"a24b4233-7e81-47f5-90b3-74601c85022b\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") "
Nov 03 20:05:01.108467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.108025    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-socket\") pod \"a24b4233-7e81-47f5-90b3-74601c85022b\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") "
Nov 03 20:05:01.108467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.108244    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-istio-token\" (UniqueName: \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-istio-token\") pod \"a24b4233-7e81-47f5-90b3-74601c85022b\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") "
Nov 03 20:05:01.108467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.108287    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a24b4233-7e81-47f5-90b3-74601c85022b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24b4233-7e81-47f5-90b3-74601c85022b-istiod-ca-cert\") pod \"a24b4233-7e81-47f5-90b3-74601c85022b\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") "
Nov 03 20:05:01.108467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.108362    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a24b4233-7e81-47f5-90b3-74601c85022b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24b4233-7e81-47f5-90b3-74601c85022b-istio-podinfo\") pod \"a24b4233-7e81-47f5-90b3-74601c85022b\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") "
Nov 03 20:05:01.108467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.108395    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-kube-api-access-phqb4\" (UniqueName: \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-kube-api-access-phqb4\") pod \"a24b4233-7e81-47f5-90b3-74601c85022b\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") "
Nov 03 20:05:01.109007 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.108490    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-envoy\") pod \"a24b4233-7e81-47f5-90b3-74601c85022b\" (UID: \"a24b4233-7e81-47f5-90b3-74601c85022b\") "
Nov 03 20:05:01.109967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.109399    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"]
Nov 03 20:05:01.110104 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.110071    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24b4233-7e81-47f5-90b3-74601c85022b-istiod-ca-cert" pod "a24b4233-7e81-47f5-90b3-74601c85022b" (UID: "a24b4233-7e81-47f5-90b3-74601c85022b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:01.110719 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.110696    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-certs" pod "a24b4233-7e81-47f5-90b3-74601c85022b" (UID: "a24b4233-7e81-47f5-90b3-74601c85022b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:01.110968 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.110956    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-credential-socket" pod "a24b4233-7e81-47f5-90b3-74601c85022b" (UID: "a24b4233-7e81-47f5-90b3-74601c85022b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:01.111902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.111883    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-data" pod "a24b4233-7e81-47f5-90b3-74601c85022b" (UID: "a24b4233-7e81-47f5-90b3-74601c85022b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:01.112217 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.112199    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-socket" pod "a24b4233-7e81-47f5-90b3-74601c85022b" (UID: "a24b4233-7e81-47f5-90b3-74601c85022b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:01.112842 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.112819    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-istio-token" pod "a24b4233-7e81-47f5-90b3-74601c85022b" (UID: "a24b4233-7e81-47f5-90b3-74601c85022b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:01.114512 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.114477    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-envoy" pod "a24b4233-7e81-47f5-90b3-74601c85022b" (UID: "a24b4233-7e81-47f5-90b3-74601c85022b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:01.114600 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.114553    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a24b4233-7e81-47f5-90b3-74601c85022b-istio-podinfo" pod "a24b4233-7e81-47f5-90b3-74601c85022b" (UID: "a24b4233-7e81-47f5-90b3-74601c85022b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:05:01.115631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.115614    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fcddeb-kdkgx"]
Nov 03 20:05:01.116037 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.116017    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-kube-api-access-phqb4" pod "a24b4233-7e81-47f5-90b3-74601c85022b" (UID: "a24b4233-7e81-47f5-90b3-74601c85022b"). InnerVolumeSpecName "kube-api-access-phqb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:01.147490 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.147454    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:01.162058 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.162030    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 20:05:01.211011 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.210961    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.211011 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.211000    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a24b4233-7e81-47f5-90b3-74601c85022b-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.211011 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.211015    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a24b4233-7e81-47f5-90b3-74601c85022b-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.211374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.211028    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phqb4\" (UniqueName: \"kubernetes.io/projected/a24b4233-7e81-47f5-90b3-74601c85022b-kube-api-access-phqb4\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.211374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.211043    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.211374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.211056    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.211374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.211071    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.211374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.211083    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.211374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.211097    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a24b4233-7e81-47f5-90b3-74601c85022b-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.335201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.332346    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-maas-api-tls\") pod \"211d2b1f-7806-4f29-8c89-35f05f3cd673\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") "
Nov 03 20:05:01.335201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.332715    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211d2b1f-7806-4f29-8c89-35f05f3cd673-kube-api-access-6gqks\" (UniqueName: \"kubernetes.io/projected/211d2b1f-7806-4f29-8c89-35f05f3cd673-kube-api-access-6gqks\") pod \"211d2b1f-7806-4f29-8c89-35f05f3cd673\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") "
Nov 03 20:05:01.335201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.332790    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211d2b1f-7806-4f29-8c89-35f05f3cd673-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211d2b1f-7806-4f29-8c89-35f05f3cd673-openshift-service-ca\") pod \"211d2b1f-7806-4f29-8c89-35f05f3cd673\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") "
Nov 03 20:05:01.335201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.332854    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-metrics-certs\" (UniqueName: \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-metrics-certs\") pod \"211d2b1f-7806-4f29-8c89-35f05f3cd673\" (UID: \"211d2b1f-7806-4f29-8c89-35f05f3cd673\") "
Nov 03 20:05:01.337458 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.337423    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211d2b1f-7806-4f29-8c89-35f05f3cd673-openshift-service-ca" pod "211d2b1f-7806-4f29-8c89-35f05f3cd673" (UID: "211d2b1f-7806-4f29-8c89-35f05f3cd673"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:01.337659 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.337586    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:01.350252 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.341140    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211d2b1f-7806-4f29-8c89-35f05f3cd673-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.350978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.350941    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-maas-api-tls" pod "211d2b1f-7806-4f29-8c89-35f05f3cd673" (UID: "211d2b1f-7806-4f29-8c89-35f05f3cd673"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:01.350978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.350961    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-metrics-certs" pod "211d2b1f-7806-4f29-8c89-35f05f3cd673" (UID: "211d2b1f-7806-4f29-8c89-35f05f3cd673"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:01.354893 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.354859    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211d2b1f-7806-4f29-8c89-35f05f3cd673-kube-api-access-6gqks" pod "211d2b1f-7806-4f29-8c89-35f05f3cd673" (UID: "211d2b1f-7806-4f29-8c89-35f05f3cd673"). InnerVolumeSpecName "kube-api-access-6gqks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:01.371646 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.371569    4531 generic.go:356] "Generic (PLEG): container finished" podID="211d2b1f-7806-4f29-8c89-35f05f3cd673" containerID="d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13" exitCode=0
Nov 03 20:05:01.371646 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.371614    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"
Nov 03 20:05:01.371824 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.371649    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr" event={"ID":"211d2b1f-7806-4f29-8c89-35f05f3cd673","Type":"ContainerDied","Data":"d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13"}
Nov 03 20:05:01.371824 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.371699    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr" event={"ID":"211d2b1f-7806-4f29-8c89-35f05f3cd673","Type":"ContainerDied","Data":"9aa09a1b27dce6f9fdfdd7e81d0eb2b138f6ff381c525f1a9702129db9d0e28c"}
Nov 03 20:05:01.371824 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.371723    4531 scope.go:122] "RemoveContainer" containerID="d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13"
Nov 03 20:05:01.373484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.373457    4531 generic.go:356] "Generic (PLEG): container finished" podID="a24b4233-7e81-47f5-90b3-74601c85022b" containerID="b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71" exitCode=0
Nov 03 20:05:01.373611 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.373510    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"
Nov 03 20:05:01.373611 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.373516    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" event={"ID":"a24b4233-7e81-47f5-90b3-74601c85022b","Type":"ContainerDied","Data":"b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71"}
Nov 03 20:05:01.373611 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.373555    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r" event={"ID":"a24b4233-7e81-47f5-90b3-74601c85022b","Type":"ContainerDied","Data":"d7c84ac34c006b74f51b4fede126657b770f1b9597329f822733768f4d3ace0c"}
Nov 03 20:05:01.383712 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.383694    4531 scope.go:122] "RemoveContainer" containerID="d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13"
Nov 03 20:05:01.384017 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:01.383995    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13\": container with ID starting with d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13 not found: ID does not exist" containerID="d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13"
Nov 03 20:05:01.384105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.384028    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13"} err="failed to get container status \"d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13\": rpc error: code = NotFound desc = could not find container \"d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13\": container with ID starting with d56d4c8b19db54988a0b5c11b86f52fc5afd9c7072b2c185dfa98ccdcb396f13 not found: ID does not exist"
Nov 03 20:05:01.384105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.384050    4531 scope.go:122] "RemoveContainer" containerID="b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71"
Nov 03 20:05:01.398706 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.398681    4531 scope.go:122] "RemoveContainer" containerID="b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71"
Nov 03 20:05:01.399331 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:01.399304    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71\": container with ID starting with b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71 not found: ID does not exist" containerID="b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71"
Nov 03 20:05:01.399438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.399335    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71"} err="failed to get container status \"b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71\": rpc error: code = NotFound desc = could not find container \"b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71\": container with ID starting with b49c54a50cafb0a8a5bb76285730c34964c3e8580b088c1e2cdad9b1d319ac71 not found: ID does not exist"
Nov 03 20:05:01.404954 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.404935    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"]
Nov 03 20:05:01.409373 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.409315    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-vdx7r"]
Nov 03 20:05:01.413608 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.413587    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"]
Nov 03 20:05:01.417377 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.417357    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fcddeb-69c698f44f-9tkcr"]
Nov 03 20:05:01.442507 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.442481    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqks\" (UniqueName: \"kubernetes.io/projected/211d2b1f-7806-4f29-8c89-35f05f3cd673-kube-api-access-6gqks\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.442507 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.442507    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.442657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.442518    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211d2b1f-7806-4f29-8c89-35f05f3cd673-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.544018 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:01.543983    4531 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-w3-32a584: configmap "payload-processing-plugins-e2e-shared-b-w3-32a584" not found
Nov 03 20:05:01.544450 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:01.544072    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume podName:6c5f6c12-d596-48e5-90bc-cc0f523a106b nodeName:}" failed. No retries permitted until 2026-11-03 20:05:03.544050765 +0000 UTC m=+2016.814174054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume") pod "payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk" (UID: "6c5f6c12-d596-48e5-90bc-cc0f523a106b") : configmap "payload-processing-plugins-e2e-shared-b-w3-32a584" not found
Nov 03 20:05:01.608599 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:01.608551    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:02.140193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.137961    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:02.551609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.551500    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8" podUID="9b84f64d-2622-4d11-aed6-ad27e8b50125" containerName="payload-processing" containerID="cri-o://3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2" gracePeriod=25
Nov 03 20:05:02.562325 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.562273    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd" podUID="00a4e015-a44e-4628-9f0c-101b362975cb" containerName="payload-pre-processing" containerID="cri-o://445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39" gracePeriod=25
Nov 03 20:05:02.850279 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.850245    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 20:05:02.863573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.863542    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211d2b1f-7806-4f29-8c89-35f05f3cd673" path="/var/lib/kubelet/pods/211d2b1f-7806-4f29-8c89-35f05f3cd673/volumes"
Nov 03 20:05:02.864405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.864369    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b84f64d-2622-4d11-aed6-ad27e8b50125-kube-api-access-s4z48\" (UniqueName: \"kubernetes.io/projected/9b84f64d-2622-4d11-aed6-ad27e8b50125-kube-api-access-s4z48\") pod \"9b84f64d-2622-4d11-aed6-ad27e8b50125\" (UID: \"9b84f64d-2622-4d11-aed6-ad27e8b50125\") "
Nov 03 20:05:02.865228 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.865015    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220d5899-b75d-4842-b3d4-e7caa9d251c5" path="/var/lib/kubelet/pods/220d5899-b75d-4842-b3d4-e7caa9d251c5/volumes"
Nov 03 20:05:02.865761 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.865721    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a24b4233-7e81-47f5-90b3-74601c85022b" path="/var/lib/kubelet/pods/a24b4233-7e81-47f5-90b3-74601c85022b/volumes"
Nov 03 20:05:02.868451 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.868424    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b84f64d-2622-4d11-aed6-ad27e8b50125-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b84f64d-2622-4d11-aed6-ad27e8b50125-plugins-config-volume\") pod \"9b84f64d-2622-4d11-aed6-ad27e8b50125\" (UID: \"9b84f64d-2622-4d11-aed6-ad27e8b50125\") "
Nov 03 20:05:02.870384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.870095    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b84f64d-2622-4d11-aed6-ad27e8b50125-plugins-config-volume" pod "9b84f64d-2622-4d11-aed6-ad27e8b50125" (UID: "9b84f64d-2622-4d11-aed6-ad27e8b50125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:02.872736 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.872714    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b84f64d-2622-4d11-aed6-ad27e8b50125-kube-api-access-s4z48" pod "9b84f64d-2622-4d11-aed6-ad27e8b50125" (UID: "9b84f64d-2622-4d11-aed6-ad27e8b50125"). InnerVolumeSpecName "kube-api-access-s4z48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:02.951522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.950940    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:02.971231 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.970797    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b84f64d-2622-4d11-aed6-ad27e8b50125-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:02.971231 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.970838    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4z48\" (UniqueName: \"kubernetes.io/projected/9b84f64d-2622-4d11-aed6-ad27e8b50125-kube-api-access-s4z48\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:02.981706 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:02.981674    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 20:05:03.071800 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.071716    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a4e015-a44e-4628-9f0c-101b362975cb-kube-api-access-6m6sk\" (UniqueName: \"kubernetes.io/projected/00a4e015-a44e-4628-9f0c-101b362975cb-kube-api-access-6m6sk\") pod \"00a4e015-a44e-4628-9f0c-101b362975cb\" (UID: \"00a4e015-a44e-4628-9f0c-101b362975cb\") "
Nov 03 20:05:03.073871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.073850    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a4e015-a44e-4628-9f0c-101b362975cb-kube-api-access-6m6sk" pod "00a4e015-a44e-4628-9f0c-101b362975cb" (UID: "00a4e015-a44e-4628-9f0c-101b362975cb"). InnerVolumeSpecName "kube-api-access-6m6sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:03.172945 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.172914    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a4e015-a44e-4628-9f0c-101b362975cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a4e015-a44e-4628-9f0c-101b362975cb-plugins-config-volume\") pod \"00a4e015-a44e-4628-9f0c-101b362975cb\" (UID: \"00a4e015-a44e-4628-9f0c-101b362975cb\") "
Nov 03 20:05:03.173377 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.173348    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a4e015-a44e-4628-9f0c-101b362975cb-plugins-config-volume" pod "00a4e015-a44e-4628-9f0c-101b362975cb" (UID: "00a4e015-a44e-4628-9f0c-101b362975cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:03.173654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.173636    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m6sk\" (UniqueName: \"kubernetes.io/projected/00a4e015-a44e-4628-9f0c-101b362975cb-kube-api-access-6m6sk\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:03.173739 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.173659    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a4e015-a44e-4628-9f0c-101b362975cb-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:03.393885 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.393845    4531 generic.go:356] "Generic (PLEG): container finished" podID="9b84f64d-2622-4d11-aed6-ad27e8b50125" containerID="3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2" exitCode=0
Nov 03 20:05:03.394108 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.393896    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"
Nov 03 20:05:03.394108 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.393927    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8" event={"ID":"9b84f64d-2622-4d11-aed6-ad27e8b50125","Type":"ContainerDied","Data":"3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2"}
Nov 03 20:05:03.394108 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.393979    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8" event={"ID":"9b84f64d-2622-4d11-aed6-ad27e8b50125","Type":"ContainerDied","Data":"9dbc0622527875a3c653fc2686db31fa197a7e91344afc4a0d37672efafef556"}
Nov 03 20:05:03.394108 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.394005    4531 scope.go:122] "RemoveContainer" containerID="3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2"
Nov 03 20:05:03.395588 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.395562    4531 generic.go:356] "Generic (PLEG): container finished" podID="00a4e015-a44e-4628-9f0c-101b362975cb" containerID="445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39" exitCode=0
Nov 03 20:05:03.395739 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.395614    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"
Nov 03 20:05:03.395739 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.395631    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd" event={"ID":"00a4e015-a44e-4628-9f0c-101b362975cb","Type":"ContainerDied","Data":"445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39"}
Nov 03 20:05:03.395739 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.395687    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd" event={"ID":"00a4e015-a44e-4628-9f0c-101b362975cb","Type":"ContainerDied","Data":"d67eca302b86249801bdd48f7a4d9d12f9d4b5804846b8b8a3f9c2acedbd3acf"}
Nov 03 20:05:03.406978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.406957    4531 scope.go:122] "RemoveContainer" containerID="3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2"
Nov 03 20:05:03.407340 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:03.407319    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2\": container with ID starting with 3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2 not found: ID does not exist" containerID="3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2"
Nov 03 20:05:03.407411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.407351    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2"} err="failed to get container status \"3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2\": rpc error: code = NotFound desc = could not find container \"3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2\": container with ID starting with 3ca31ba631dcbc480fe730c8e3aab476e5b4fb0671ef2cff17cd6f8ec7cea4a2 not found: ID does not exist"
Nov 03 20:05:03.407411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.407367    4531 scope.go:122] "RemoveContainer" containerID="445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39"
Nov 03 20:05:03.417841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.417821    4531 scope.go:122] "RemoveContainer" containerID="445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39"
Nov 03 20:05:03.418154 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:03.418134    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39\": container with ID starting with 445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39 not found: ID does not exist" containerID="445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39"
Nov 03 20:05:03.418237 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.418161    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39"} err="failed to get container status \"445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39\": rpc error: code = NotFound desc = could not find container \"445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39\": container with ID starting with 445b29758f3cac59168be9166e929d8d3a43f60f05ffa562d745994b8b349e39 not found: ID does not exist"
Nov 03 20:05:03.420681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.420657    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"]
Nov 03 20:05:03.426094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.426067    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fcddeb-66f88c7459-zrsmd"]
Nov 03 20:05:03.434247 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.434223    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"]
Nov 03 20:05:03.449821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.449779    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fcddeb-7b6cccd547-cn5d8"]
Nov 03 20:05:03.572879 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.572407    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:03.599736 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:03.599209    4531 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-w3-32a584: configmap "payload-processing-plugins-e2e-shared-b-w3-32a584" not found
Nov 03 20:05:03.599736 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:03.599316    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume podName:6c5f6c12-d596-48e5-90bc-cc0f523a106b nodeName:}" failed. No retries permitted until 2026-11-03 20:05:07.599281815 +0000 UTC m=+2020.869405106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume") pod "payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk" (UID: "6c5f6c12-d596-48e5-90bc-cc0f523a106b") : configmap "payload-processing-plugins-e2e-shared-b-w3-32a584" not found
Nov 03 20:05:03.991352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:03.991316    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:04.123246 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:04.120120    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:04.322257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:04.314218    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:04.861033 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:04.861002    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a4e015-a44e-4628-9f0c-101b362975cb" path="/var/lib/kubelet/pods/00a4e015-a44e-4628-9f0c-101b362975cb/volumes"
Nov 03 20:05:04.861549 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:04.861531    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b84f64d-2622-4d11-aed6-ad27e8b50125" path="/var/lib/kubelet/pods/9b84f64d-2622-4d11-aed6-ad27e8b50125/volumes"
Nov 03 20:05:04.883381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:04.883353    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb" podUID="8f6a64a2-11cd-4280-b96a-ee0985f8ebe8" containerName="payload-processing" containerID="cri-o://336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3" gracePeriod=25
Nov 03 20:05:05.107526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.106065    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:05.223458 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.223431    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 20:05:05.333414 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.333375    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-plugins-config-volume\") pod \"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8\" (UID: \"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8\") "
Nov 03 20:05:05.333587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.333533    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-kube-api-access-tdlvh\" (UniqueName: \"kubernetes.io/projected/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-kube-api-access-tdlvh\") pod \"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8\" (UID: \"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8\") "
Nov 03 20:05:05.333782 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.333754    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-plugins-config-volume" pod "8f6a64a2-11cd-4280-b96a-ee0985f8ebe8" (UID: "8f6a64a2-11cd-4280-b96a-ee0985f8ebe8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:05.334365 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.334346    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:05.335695 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.335668    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-kube-api-access-tdlvh" pod "8f6a64a2-11cd-4280-b96a-ee0985f8ebe8" (UID: "8f6a64a2-11cd-4280-b96a-ee0985f8ebe8"). InnerVolumeSpecName "kube-api-access-tdlvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:05.363497 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.363463    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk" podUID="6c5f6c12-d596-48e5-90bc-cc0f523a106b" containerName="payload-pre-processing" containerID="cri-o://39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a" gracePeriod=25
Nov 03 20:05:05.426402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.426364    4531 generic.go:356] "Generic (PLEG): container finished" podID="8f6a64a2-11cd-4280-b96a-ee0985f8ebe8" containerID="336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3" exitCode=0
Nov 03 20:05:05.426596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.426410    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"
Nov 03 20:05:05.426596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.426440    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb" event={"ID":"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8","Type":"ContainerDied","Data":"336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3"}
Nov 03 20:05:05.426596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.426504    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb" event={"ID":"8f6a64a2-11cd-4280-b96a-ee0985f8ebe8","Type":"ContainerDied","Data":"ced4b1fb83a70420d12c4ea72746b25764d2b31edd00044f14cd2d7f86755fcc"}
Nov 03 20:05:05.426596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.426534    4531 scope.go:122] "RemoveContainer" containerID="336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3"
Nov 03 20:05:05.435998 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.435975    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlvh\" (UniqueName: \"kubernetes.io/projected/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8-kube-api-access-tdlvh\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:05.453249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.453096    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"]
Nov 03 20:05:05.456983 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.456956    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32a584-665d8875f9-dc7rb"]
Nov 03 20:05:05.477654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.477634    4531 scope.go:122] "RemoveContainer" containerID="336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3"
Nov 03 20:05:05.478018 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:05.477994    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3\": container with ID starting with 336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3 not found: ID does not exist" containerID="336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3"
Nov 03 20:05:05.478127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.478025    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3"} err="failed to get container status \"336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3\": rpc error: code = NotFound desc = could not find container \"336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3\": container with ID starting with 336bd524c899e8402a227acefe3276754f3a14d1260ed2fd34d2739f74d28be3 not found: ID does not exist"
Nov 03 20:05:05.633803 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.633778    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 20:05:05.759276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.754716    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume\") pod \"6c5f6c12-d596-48e5-90bc-cc0f523a106b\" (UID: \"6c5f6c12-d596-48e5-90bc-cc0f523a106b\") "
Nov 03 20:05:05.759276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.754900    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5f6c12-d596-48e5-90bc-cc0f523a106b-kube-api-access-ccw9t\" (UniqueName: \"kubernetes.io/projected/6c5f6c12-d596-48e5-90bc-cc0f523a106b-kube-api-access-ccw9t\") pod \"6c5f6c12-d596-48e5-90bc-cc0f523a106b\" (UID: \"6c5f6c12-d596-48e5-90bc-cc0f523a106b\") "
Nov 03 20:05:05.785223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.768796    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:05.785223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.770313    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume" pod "6c5f6c12-d596-48e5-90bc-cc0f523a106b" (UID: "6c5f6c12-d596-48e5-90bc-cc0f523a106b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:05.785223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.771334    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f6c12-d596-48e5-90bc-cc0f523a106b-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:05.785223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.773410    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5f6c12-d596-48e5-90bc-cc0f523a106b-kube-api-access-ccw9t" pod "6c5f6c12-d596-48e5-90bc-cc0f523a106b" (UID: "6c5f6c12-d596-48e5-90bc-cc0f523a106b"). InnerVolumeSpecName "kube-api-access-ccw9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:05.873606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:05.873566    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccw9t\" (UniqueName: \"kubernetes.io/projected/6c5f6c12-d596-48e5-90bc-cc0f523a106b-kube-api-access-ccw9t\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:06.180352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.180312    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:06.339195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.325763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:06.505242 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.495890    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:06.519197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.512471    4531 generic.go:356] "Generic (PLEG): container finished" podID="6c5f6c12-d596-48e5-90bc-cc0f523a106b" containerID="39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a" exitCode=0
Nov 03 20:05:06.519197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.512593    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk" event={"ID":"6c5f6c12-d596-48e5-90bc-cc0f523a106b","Type":"ContainerDied","Data":"39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a"}
Nov 03 20:05:06.519197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.512645    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk" event={"ID":"6c5f6c12-d596-48e5-90bc-cc0f523a106b","Type":"ContainerDied","Data":"366b43bfeb7c7d9783c322effc21d3c8018f9a347ac60a80c3b91daf76e13860"}
Nov 03 20:05:06.519197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.512679    4531 scope.go:122] "RemoveContainer" containerID="39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a"
Nov 03 20:05:06.519197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.512816    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"
Nov 03 20:05:06.589871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.589163    4531 scope.go:122] "RemoveContainer" containerID="39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a"
Nov 03 20:05:06.589871 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:06.589752    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a\": container with ID starting with 39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a not found: ID does not exist" containerID="39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a"
Nov 03 20:05:06.589871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.589789    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a"} err="failed to get container status \"39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a\": rpc error: code = NotFound desc = could not find container \"39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a\": container with ID starting with 39157820a77fe62c3d65831d010b3f51d8820589f01824db6259fa1fc80bdc6a not found: ID does not exist"
Nov 03 20:05:06.606849 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.606251    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"]
Nov 03 20:05:06.613011 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.612937    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32a584-546dc64c87-x9hsk"]
Nov 03 20:05:06.865405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.865323    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c5f6c12-d596-48e5-90bc-cc0f523a106b" path="/var/lib/kubelet/pods/6c5f6c12-d596-48e5-90bc-cc0f523a106b/volumes"
Nov 03 20:05:06.866042 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:06.866020    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f6a64a2-11cd-4280-b96a-ee0985f8ebe8" path="/var/lib/kubelet/pods/8f6a64a2-11cd-4280-b96a-ee0985f8ebe8/volumes"
Nov 03 20:05:07.434247 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:07.433865    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:07.614477 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:07.614449    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:07.751860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:07.751743    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:07.837723 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:07.837683    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"]
Nov 03 20:05:07.838272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:07.838240    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerName="istio-proxy" containerID="cri-o://1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9" gracePeriod=30
Nov 03 20:05:07.914537 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:07.914501    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:08.087116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.087023    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:08.319209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.317608    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:08.460194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.460133    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:08.529898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.529868    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"]
Nov 03 20:05:08.530644 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530627    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8f6a64a2-11cd-4280-b96a-ee0985f8ebe8" containerName="payload-processing"
Nov 03 20:05:08.530644 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530644    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6a64a2-11cd-4280-b96a-ee0985f8ebe8" containerName="payload-processing"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530662    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2dcebb6a-d39d-4eba-ba2c-5dab051b4dad" containerName="revoke-keys"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530667    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dcebb6a-d39d-4eba-ba2c-5dab051b4dad" containerName="revoke-keys"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530679    4531 cpu_manager.go:414] "containerMap: removing container" podUID="211d2b1f-7806-4f29-8c89-35f05f3cd673" containerName="maas-api"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530684    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211d2b1f-7806-4f29-8c89-35f05f3cd673" containerName="maas-api"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530715    4531 cpu_manager.go:414] "containerMap: removing container" podUID="00a4e015-a44e-4628-9f0c-101b362975cb" containerName="payload-pre-processing"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530721    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a4e015-a44e-4628-9f0c-101b362975cb" containerName="payload-pre-processing"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530730    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerName="istio-proxy"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530735    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerName="istio-proxy"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530742    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a24b4233-7e81-47f5-90b3-74601c85022b" containerName="istio-proxy"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530747    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24b4233-7e81-47f5-90b3-74601c85022b" containerName="istio-proxy"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530758    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9b84f64d-2622-4d11-aed6-ad27e8b50125" containerName="payload-processing"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530762    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b84f64d-2622-4d11-aed6-ad27e8b50125" containerName="payload-processing"
Nov 03 20:05:08.530766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530771    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6c5f6c12-d596-48e5-90bc-cc0f523a106b" containerName="payload-pre-processing"
Nov 03 20:05:08.531153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530777    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c5f6c12-d596-48e5-90bc-cc0f523a106b" containerName="payload-pre-processing"
Nov 03 20:05:08.531153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530885    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a4e015-a44e-4628-9f0c-101b362975cb" containerName="payload-pre-processing"
Nov 03 20:05:08.531153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530896    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edf055d-9f1e-415e-a672-8cfd8e4d2263" containerName="istio-proxy"
Nov 03 20:05:08.531153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530906    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24b4233-7e81-47f5-90b3-74601c85022b" containerName="istio-proxy"
Nov 03 20:05:08.531153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530916    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b84f64d-2622-4d11-aed6-ad27e8b50125" containerName="payload-processing"
Nov 03 20:05:08.531153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530924    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c5f6c12-d596-48e5-90bc-cc0f523a106b" containerName="payload-pre-processing"
Nov 03 20:05:08.531153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530933    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6a64a2-11cd-4280-b96a-ee0985f8ebe8" containerName="payload-processing"
Nov 03 20:05:08.531153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530941    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dcebb6a-d39d-4eba-ba2c-5dab051b4dad" containerName="revoke-keys"
Nov 03 20:05:08.531153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.530948    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="211d2b1f-7806-4f29-8c89-35f05f3cd673" containerName="maas-api"
Nov 03 20:05:08.536100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.536067    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:08.539167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.539137    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"]
Nov 03 20:05:08.539522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.539498    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:05:08.539676 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.539565    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:05:08.582603 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.582555    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2" podUID="13a23eb6-a465-4311-8dc1-3b6cc3d0571a" containerName="main" containerID="cri-o://5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56" gracePeriod=45
Nov 03 20:05:08.613080 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.613053    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:08.615181 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.615147    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:08.615292 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.615281    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:08.615379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.615366    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq8gt\" (UniqueName: \"kubernetes.io/projected/9ca5dad5-b21a-4e0d-854d-d647e27d6775-kube-api-access-tq8gt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:08.716783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.716701    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq8gt\" (UniqueName: \"kubernetes.io/projected/9ca5dad5-b21a-4e0d-854d-d647e27d6775-kube-api-access-tq8gt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:08.717006 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.716988    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:08.717116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.717100    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:08.717805 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.717759    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:08.730907 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.730873    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:08.744041 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.744005    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq8gt\" (UniqueName: \"kubernetes.io/projected/9ca5dad5-b21a-4e0d-854d-d647e27d6775-kube-api-access-tq8gt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:08.787276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.787234    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:08.917195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.916351    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:08.997860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:08.997778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:09.087035 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.087002    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 20:05:09.146747 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.146701    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-kube-api-access-l9tg7\" (UniqueName: \"kubernetes.io/projected/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-kube-api-access-l9tg7\") pod \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") "
Nov 03 20:05:09.147052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.147028    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-home\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-home\") pod \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") "
Nov 03 20:05:09.147250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.147232    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-dshm\") pod \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") "
Nov 03 20:05:09.147396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.147368    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-home" pod "13a23eb6-a465-4311-8dc1-3b6cc3d0571a" (UID: "13a23eb6-a465-4311-8dc1-3b6cc3d0571a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:09.147676 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.147654    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tls-certs\" (UniqueName: \"kubernetes.io/secret/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tls-certs\") pod \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") "
Nov 03 20:05:09.147824 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.147807    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-model-cache\") pod \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") "
Nov 03 20:05:09.148046 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.148028    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tmp-dir\") pod \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\" (UID: \"13a23eb6-a465-4311-8dc1-3b6cc3d0571a\") "
Nov 03 20:05:09.148379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.148332    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tmp-dir" pod "13a23eb6-a465-4311-8dc1-3b6cc3d0571a" (UID: "13a23eb6-a465-4311-8dc1-3b6cc3d0571a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:09.148379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.148339    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-model-cache" pod "13a23eb6-a465-4311-8dc1-3b6cc3d0571a" (UID: "13a23eb6-a465-4311-8dc1-3b6cc3d0571a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:09.149049 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.149027    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:09.149049 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.149051    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:09.149240 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.149061    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:09.150596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.150558    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-dshm" pod "13a23eb6-a465-4311-8dc1-3b6cc3d0571a" (UID: "13a23eb6-a465-4311-8dc1-3b6cc3d0571a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:09.150704 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.150589    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-kube-api-access-l9tg7" pod "13a23eb6-a465-4311-8dc1-3b6cc3d0571a" (UID: "13a23eb6-a465-4311-8dc1-3b6cc3d0571a"). InnerVolumeSpecName "kube-api-access-l9tg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:09.156964 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.155430    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tls-certs" pod "13a23eb6-a465-4311-8dc1-3b6cc3d0571a" (UID: "13a23eb6-a465-4311-8dc1-3b6cc3d0571a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:09.250673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.250554    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9tg7\" (UniqueName: \"kubernetes.io/projected/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-kube-api-access-l9tg7\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:09.250673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.250597    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:09.250673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.250614    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a23eb6-a465-4311-8dc1-3b6cc3d0571a-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:09.382007 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.381972    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"]
Nov 03 20:05:09.383463 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:05:09.383439    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca5dad5_b21a_4e0d_854d_d647e27d6775.slice/crio-db27278d29dfe8f19b24e32f9961b0611886dab89837a06fbad9a523bba78a47 WatchSource:0}: Error finding container db27278d29dfe8f19b24e32f9961b0611886dab89837a06fbad9a523bba78a47: Status 404 returned error can't find the container with id db27278d29dfe8f19b24e32f9961b0611886dab89837a06fbad9a523bba78a47
Nov 03 20:05:09.543810 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.543715    4531 generic.go:356] "Generic (PLEG): container finished" podID="13a23eb6-a465-4311-8dc1-3b6cc3d0571a" containerID="5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56" exitCode=0
Nov 03 20:05:09.543810 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.543769    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"
Nov 03 20:05:09.543810 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.543791    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2" event={"ID":"13a23eb6-a465-4311-8dc1-3b6cc3d0571a","Type":"ContainerDied","Data":"5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56"}
Nov 03 20:05:09.544387 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.543834    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2" event={"ID":"13a23eb6-a465-4311-8dc1-3b6cc3d0571a","Type":"ContainerDied","Data":"6f9e8c23da2f3c1b7dc6cf6858120ac7797a65f384cef224d69f5b51d47b346e"}
Nov 03 20:05:09.544387 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.543860    4531 scope.go:122] "RemoveContainer" containerID="5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56"
Nov 03 20:05:09.545685 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.545653    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs" event={"ID":"9ca5dad5-b21a-4e0d-854d-d647e27d6775","Type":"ContainerStarted","Data":"db27278d29dfe8f19b24e32f9961b0611886dab89837a06fbad9a523bba78a47"}
Nov 03 20:05:09.556842 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.556819    4531 scope.go:122] "RemoveContainer" containerID="5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56"
Nov 03 20:05:09.557102 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:09.557080    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56\": container with ID starting with 5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56 not found: ID does not exist" containerID="5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56"
Nov 03 20:05:09.557148 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.557111    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56"} err="failed to get container status \"5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56\": rpc error: code = NotFound desc = could not find container \"5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56\": container with ID starting with 5fb253f30b8b452c53f214723c431556b27f76a8d3ef3fc24cb96d9d8579ef56 not found: ID does not exist"
Nov 03 20:05:09.574873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.574848    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"]
Nov 03 20:05:09.579889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.579864    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c18bea/test-model-c18bea-kserve-9cd6df8f5-ts6w2"]
Nov 03 20:05:09.652929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:09.652894    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:10.359200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:10.355727    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:10.557099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:10.557067    4531 generic.go:356] "Generic (PLEG): container finished" podID="9ca5dad5-b21a-4e0d-854d-d647e27d6775" containerID="a097922824281914bcc21724af7047f53f1c4a896f812a582c0950e9ab41474d" exitCode=0
Nov 03 20:05:10.557530 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:10.557152    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs" event={"ID":"9ca5dad5-b21a-4e0d-854d-d647e27d6775","Type":"ContainerDied","Data":"a097922824281914bcc21724af7047f53f1c4a896f812a582c0950e9ab41474d"}
Nov 03 20:05:10.862518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:10.862484    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a23eb6-a465-4311-8dc1-3b6cc3d0571a" path="/var/lib/kubelet/pods/13a23eb6-a465-4311-8dc1-3b6cc3d0571a/volumes"
Nov 03 20:05:11.027834 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:11.020842    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:11.760198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:11.757731    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:11.782623 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:11.782592    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:11.901899 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:11.901862    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-xks-ca\") pod \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") "
Nov 03 20:05:11.902098 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:11.901932    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca5dad5-b21a-4e0d-854d-d647e27d6775-kube-api-access-tq8gt\" (UniqueName: \"kubernetes.io/projected/9ca5dad5-b21a-4e0d-854d-d647e27d6775-kube-api-access-tq8gt\") pod \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") "
Nov 03 20:05:11.902098 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:11.902003    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-openshift-ca\") pod \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\" (UID: \"9ca5dad5-b21a-4e0d-854d-d647e27d6775\") "
Nov 03 20:05:11.902450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:11.902425    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-openshift-ca" pod "9ca5dad5-b21a-4e0d-854d-d647e27d6775" (UID: "9ca5dad5-b21a-4e0d-854d-d647e27d6775"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:11.902778 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:11.902749    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:11.904132 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:11.904110    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-xks-ca" pod "9ca5dad5-b21a-4e0d-854d-d647e27d6775" (UID: "9ca5dad5-b21a-4e0d-854d-d647e27d6775"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:11.904482 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:11.904463    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca5dad5-b21a-4e0d-854d-d647e27d6775-kube-api-access-tq8gt" pod "9ca5dad5-b21a-4e0d-854d-d647e27d6775" (UID: "9ca5dad5-b21a-4e0d-854d-d647e27d6775"). InnerVolumeSpecName "kube-api-access-tq8gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:12.004244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:12.004168    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca5dad5-b21a-4e0d-854d-d647e27d6775-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:12.004244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:12.004245    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq8gt\" (UniqueName: \"kubernetes.io/projected/9ca5dad5-b21a-4e0d-854d-d647e27d6775-kube-api-access-tq8gt\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:12.450203 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:12.444741    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:12.575705 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:12.575677    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"
Nov 03 20:05:12.575898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:12.575710    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs" event={"ID":"9ca5dad5-b21a-4e0d-854d-d647e27d6775","Type":"ContainerDied","Data":"db27278d29dfe8f19b24e32f9961b0611886dab89837a06fbad9a523bba78a47"}
Nov 03 20:05:12.575898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:12.575758    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db27278d29dfe8f19b24e32f9961b0611886dab89837a06fbad9a523bba78a47"
Nov 03 20:05:12.977264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:12.977230    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:13.260196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.255390    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:13.379732 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.379701    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 20:05:13.406660 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.406611    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:13.462431 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.462398    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-podinfo\") pod \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") "
Nov 03 20:05:13.462819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.462478    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-socket\") pod \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") "
Nov 03 20:05:13.462819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.462514    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-kube-api-access-2h4dm\" (UniqueName: \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-kube-api-access-2h4dm\") pod \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") "
Nov 03 20:05:13.462819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.462567    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-envoy\") pod \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") "
Nov 03 20:05:13.462819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.462644    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-token\" (UniqueName: \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-token\") pod \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") "
Nov 03 20:05:13.463145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.462836    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-credential-socket\") pod \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") "
Nov 03 20:05:13.463145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.462931    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-data\") pod \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") "
Nov 03 20:05:13.463145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.462970    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-certs\") pod \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") "
Nov 03 20:05:13.463145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.463003    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istiod-ca-cert\") pod \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\" (UID: \"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd\") "
Nov 03 20:05:13.463470 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.463154    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-socket" pod "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" (UID: "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:13.463470 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.463439    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-data" pod "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" (UID: "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:13.463903 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.463781    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istiod-ca-cert" pod "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" (UID: "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:13.464017 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.464002    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.464077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.464027    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.464077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.464044    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.464203 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.464142    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-credential-socket" pod "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" (UID: "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:13.464275 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.464243    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-certs" pod "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" (UID: "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:13.465143 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.465122    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-kube-api-access-2h4dm" pod "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" (UID: "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd"). InnerVolumeSpecName "kube-api-access-2h4dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:13.465266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.465153    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-token" pod "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" (UID: "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:13.465266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.465230    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-podinfo" pod "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" (UID: "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:05:13.465266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.465242    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-envoy" pod "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" (UID: "b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:13.555224 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.548270    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:13.589927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.589829    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h4dm\" (UniqueName: \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-kube-api-access-2h4dm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.589927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.589863    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.589927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.589879    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.589927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.589893    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.589927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.589906    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.589927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.589919    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.599075 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.599040    4531 generic.go:356] "Generic (PLEG): container finished" podID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerID="1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9" exitCode=0
Nov 03 20:05:13.599311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.599091    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"
Nov 03 20:05:13.599311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.599101    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" event={"ID":"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd","Type":"ContainerDied","Data":"1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9"}
Nov 03 20:05:13.599311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.599153    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx" event={"ID":"b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd","Type":"ContainerDied","Data":"fecf34ff898fdbc877e1327ddb5417c490e42109b13323e960bc3d0ceac7c0bf"}
Nov 03 20:05:13.599311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.599197    4531 scope.go:122] "RemoveContainer" containerID="1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9"
Nov 03 20:05:13.611443 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.611423    4531 scope.go:122] "RemoveContainer" containerID="1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9"
Nov 03 20:05:13.611713 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:13.611693    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9\": container with ID starting with 1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9 not found: ID does not exist" containerID="1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9"
Nov 03 20:05:13.611793 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.611719    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9"} err="failed to get container status \"1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9\": rpc error: code = NotFound desc = could not find container \"1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9\": container with ID starting with 1933d8091da9d1d27542e43c3f71737a397a2fbef747162620c07332b0a37ba9 not found: ID does not exist"
Nov 03 20:05:13.626824 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.626784    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"]
Nov 03 20:05:13.633021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.632987    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nfjvx"]
Nov 03 20:05:13.683459 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:13.683426    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:14.400771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:14.399219    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:14.860097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:14.860011    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" path="/var/lib/kubelet/pods/b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd/volumes"
Nov 03 20:05:15.001372 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.001338    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:15.164815 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.164780    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:15.316195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.310660    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:15.316195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.315955    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"]
Nov 03 20:05:15.339280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.322900    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3cee6a-nc65n"]
Nov 03 20:05:15.450871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.450787    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:15.585630 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.585593    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:15.755231 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.755099    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:15.838989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.838956    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"]
Nov 03 20:05:15.839706 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.839690    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9ca5dad5-b21a-4e0d-854d-d647e27d6775" containerName="revoke-keys"
Nov 03 20:05:15.839783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.839707    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca5dad5-b21a-4e0d-854d-d647e27d6775" containerName="revoke-keys"
Nov 03 20:05:15.839783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.839739    4531 cpu_manager.go:414] "containerMap: removing container" podUID="13a23eb6-a465-4311-8dc1-3b6cc3d0571a" containerName="main"
Nov 03 20:05:15.839783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.839745    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a23eb6-a465-4311-8dc1-3b6cc3d0571a" containerName="main"
Nov 03 20:05:15.839783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.839765    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerName="istio-proxy"
Nov 03 20:05:15.839783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.839771    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerName="istio-proxy"
Nov 03 20:05:15.839943 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.839912    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca5dad5-b21a-4e0d-854d-d647e27d6775" containerName="revoke-keys"
Nov 03 20:05:15.839943 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.839925    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a23eb6-a465-4311-8dc1-3b6cc3d0571a" containerName="main"
Nov 03 20:05:15.839943 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.839934    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0edc976-7bb9-4cdf-82d4-cdd3ba9c92fd" containerName="istio-proxy"
Nov 03 20:05:15.845433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.845415    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:15.849279 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.849250    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-3ff278-openshift-default-dockercfg-c45jl\""
Nov 03 20:05:15.856752 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.856728    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"]
Nov 03 20:05:15.927264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.927231    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:15.950906 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.950869    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-certs\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:15.951161 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.951129    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9e6662-96f2-42b0-853a-9649585967b5-istiod-ca-cert\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:15.951415 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.951388    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-istio-token\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:15.951529 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.951499    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-data\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:15.951621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.951524    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9e6662-96f2-42b0-853a-9649585967b5-istio-podinfo\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:15.951621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.951561    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-envoy\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:15.951733 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.951631    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:15.951786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.951770    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq2pp\" (UniqueName: \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-kube-api-access-rq2pp\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:15.951839 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:15.951820    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-credential-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.018039 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.017960    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"]
Nov 03 20:05:16.022511 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.022491    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:16.028707 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.028679    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"]
Nov 03 20:05:16.050003 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.049977    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:05:16.050186 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.050086    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:05:16.053088 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053061    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-data\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.053295 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053104    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9e6662-96f2-42b0-853a-9649585967b5-istio-podinfo\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.053295 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053139    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-envoy\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.053295 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053222    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.053465 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053344    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq2pp\" (UniqueName: \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-kube-api-access-rq2pp\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.053465 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053374    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-credential-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.053465 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053448    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-certs\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.053627 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053583    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-data\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.053627 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053585    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9e6662-96f2-42b0-853a-9649585967b5-istiod-ca-cert\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.053735 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053635    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.053735 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053714    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-istio-token\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.053934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.053903    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-credential-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.054352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.054329    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9e6662-96f2-42b0-853a-9649585967b5-istiod-ca-cert\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.054352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.054348    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-certs\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.056589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.056562    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9e6662-96f2-42b0-853a-9649585967b5-istio-podinfo\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.056680 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.056606    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-envoy\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.060981 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.060953    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-istio-token\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.061112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.061090    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq2pp\" (UniqueName: \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-kube-api-access-rq2pp\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.155029 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.154980    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:16.155293 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.155049    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcv6q\" (UniqueName: \"kubernetes.io/projected/24716c14-b72e-4d1e-b51d-34b1f587bd92-kube-api-access-hcv6q\") pod \"maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:16.155410 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.155289    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:16.163523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.163486    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:16.259489 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.257481    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:16.259489 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.257558    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcv6q\" (UniqueName: \"kubernetes.io/projected/24716c14-b72e-4d1e-b51d-34b1f587bd92-kube-api-access-hcv6q\") pod \"maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:16.259489 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.257745    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:16.259489 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.258814    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:16.262223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.261727    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:16.266911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.266883    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcv6q\" (UniqueName: \"kubernetes.io/projected/24716c14-b72e-4d1e-b51d-34b1f587bd92-kube-api-access-hcv6q\") pod \"maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:16.344382 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.344305    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:16.457618 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.457578    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"]
Nov 03 20:05:16.464942 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:05:16.464910    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9e6662_96f2_42b0_853a_9649585967b5.slice/crio-7956992ffb46031e30d9e75e73a5f6b820a8e1130289e934a3965159d7a8f5f5 WatchSource:0}: Error finding container 7956992ffb46031e30d9e75e73a5f6b820a8e1130289e934a3965159d7a8f5f5: Status 404 returned error can't find the container with id 7956992ffb46031e30d9e75e73a5f6b820a8e1130289e934a3965159d7a8f5f5
Nov 03 20:05:16.468186 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.468128    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:16.468294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.468243    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:16.468370 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.468290    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:16.633166 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.624035    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:16.659912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.659870    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"]
Nov 03 20:05:16.705893 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:05:16.705471    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24716c14_b72e_4d1e_b51d_34b1f587bd92.slice/crio-93c8a011ded4081f6e0ce095ba9f1ae08e6bd785cfea78906ea4fdaa371545e8 WatchSource:0}: Error finding container 93c8a011ded4081f6e0ce095ba9f1ae08e6bd785cfea78906ea4fdaa371545e8: Status 404 returned error can't find the container with id 93c8a011ded4081f6e0ce095ba9f1ae08e6bd785cfea78906ea4fdaa371545e8
Nov 03 20:05:16.715234 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.715199    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" event={"ID":"8d9e6662-96f2-42b0-853a-9649585967b5","Type":"ContainerStarted","Data":"48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65"}
Nov 03 20:05:16.715322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.715270    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" event={"ID":"8d9e6662-96f2-42b0-853a-9649585967b5","Type":"ContainerStarted","Data":"7956992ffb46031e30d9e75e73a5f6b820a8e1130289e934a3965159d7a8f5f5"}
Nov 03 20:05:16.735656 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.735612    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" podStartSLOduration=1.735599416 podStartE2EDuration="1.735599416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:16.73051117 +0000 UTC m=+2030.000634465" watchObservedRunningTime="2026-11-03 20:05:16.735599416 +0000 UTC m=+2030.005722709"
Nov 03 20:05:16.862147 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:16.862096    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dcebb6a-d39d-4eba-ba2c-5dab051b4dad" path="/var/lib/kubelet/pods/2dcebb6a-d39d-4eba-ba2c-5dab051b4dad/volumes"
Nov 03 20:05:17.063259 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:17.061809    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:17.164606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:17.164558    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:17.166320 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:17.166289    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:05:17.166439 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:17.166349    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" podUID="8d9e6662-96f2-42b0-853a-9649585967b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 20:05:17.596196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:17.593657    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:17.729539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:17.729491    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv" event={"ID":"24716c14-b72e-4d1e-b51d-34b1f587bd92","Type":"ContainerStarted","Data":"6d2d7bfb189aa1398bf761ede5e896f3aba8197bd0e313850e4214ab6c949e9b"}
Nov 03 20:05:17.729758 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:17.729601    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv" event={"ID":"24716c14-b72e-4d1e-b51d-34b1f587bd92","Type":"ContainerStarted","Data":"93c8a011ded4081f6e0ce095ba9f1ae08e6bd785cfea78906ea4fdaa371545e8"}
Nov 03 20:05:17.766695 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:17.766630    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv" podStartSLOduration=1.766614594 podStartE2EDuration="1.766614594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:17.764448409 +0000 UTC m=+2031.034571703" watchObservedRunningTime="2026-11-03 20:05:17.766614594 +0000 UTC m=+2031.036737888"
Nov 03 20:05:18.164725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:18.164681    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:05:18.164923 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:18.164757    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" podUID="8d9e6662-96f2-42b0-853a-9649585967b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 20:05:18.277193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:18.273560    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:18.758197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:18.751356    4531 generic.go:356] "Generic (PLEG): container finished" podID="24716c14-b72e-4d1e-b51d-34b1f587bd92" containerID="6d2d7bfb189aa1398bf761ede5e896f3aba8197bd0e313850e4214ab6c949e9b" exitCode=0
Nov 03 20:05:18.758197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:18.752289    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv" event={"ID":"24716c14-b72e-4d1e-b51d-34b1f587bd92","Type":"ContainerDied","Data":"6d2d7bfb189aa1398bf761ede5e896f3aba8197bd0e313850e4214ab6c949e9b"}
Nov 03 20:05:18.771228 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:18.771189    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"]
Nov 03 20:05:18.787336 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:18.787288    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"]
Nov 03 20:05:18.891574 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:18.891518    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:19.027417 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:19.027342    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:19.166370 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:19.166329    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:05:19.166606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:19.166417    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" podUID="8d9e6662-96f2-42b0-853a-9649585967b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 20:05:19.167778 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:19.167749    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:19.876194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:19.859728    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:20.006930 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.006803    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:20.072261 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.072235    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:20.111067 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.111025    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-openshift-ca\") pod \"24716c14-b72e-4d1e-b51d-34b1f587bd92\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") "
Nov 03 20:05:20.111340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.111092    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-xks-ca\") pod \"24716c14-b72e-4d1e-b51d-34b1f587bd92\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") "
Nov 03 20:05:20.112304 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.112261    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-openshift-ca" pod "24716c14-b72e-4d1e-b51d-34b1f587bd92" (UID: "24716c14-b72e-4d1e-b51d-34b1f587bd92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:20.117539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.117514    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"]
Nov 03 20:05:20.117932 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.117905    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy" containerID="cri-o://a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2" gracePeriod=30
Nov 03 20:05:20.119652 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.119626    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-xks-ca" pod "24716c14-b72e-4d1e-b51d-34b1f587bd92" (UID: "24716c14-b72e-4d1e-b51d-34b1f587bd92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:20.171232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.166767    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:05:20.171232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.166827    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" podUID="8d9e6662-96f2-42b0-853a-9649585967b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 20:05:20.196221 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.196159    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:20.212843 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.212805    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24716c14-b72e-4d1e-b51d-34b1f587bd92-kube-api-access-hcv6q\" (UniqueName: \"kubernetes.io/projected/24716c14-b72e-4d1e-b51d-34b1f587bd92-kube-api-access-hcv6q\") pod \"24716c14-b72e-4d1e-b51d-34b1f587bd92\" (UID: \"24716c14-b72e-4d1e-b51d-34b1f587bd92\") "
Nov 03 20:05:20.213667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.213638    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:20.213667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.213667    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24716c14-b72e-4d1e-b51d-34b1f587bd92-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:20.218322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.218272    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24716c14-b72e-4d1e-b51d-34b1f587bd92-kube-api-access-hcv6q" pod "24716c14-b72e-4d1e-b51d-34b1f587bd92" (UID: "24716c14-b72e-4d1e-b51d-34b1f587bd92"). InnerVolumeSpecName "kube-api-access-hcv6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:20.324250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.323621    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcv6q\" (UniqueName: \"kubernetes.io/projected/24716c14-b72e-4d1e-b51d-34b1f587bd92-kube-api-access-hcv6q\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:20.592337 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:20.592245    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:05:20.633638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.633604    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"]
Nov 03 20:05:20.634722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.634695    4531 cpu_manager.go:414] "containerMap: removing container" podUID="24716c14-b72e-4d1e-b51d-34b1f587bd92" containerName="revoke-keys"
Nov 03 20:05:20.634722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.634720    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24716c14-b72e-4d1e-b51d-34b1f587bd92" containerName="revoke-keys"
Nov 03 20:05:20.634948 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.634935    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="24716c14-b72e-4d1e-b51d-34b1f587bd92" containerName="revoke-keys"
Nov 03 20:05:20.643208 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.642525    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:20.731904 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.724661    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:20.754272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.754232    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:20.754720 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.754691    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:20.754976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.754954    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxjtj\" (UniqueName: \"kubernetes.io/projected/b9096d4f-4e10-4310-b688-6c7e64c371f7-kube-api-access-gxjtj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:20.773064 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.773035    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"]
Nov 03 20:05:20.806715 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.806673    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv" event={"ID":"24716c14-b72e-4d1e-b51d-34b1f587bd92","Type":"ContainerDied","Data":"93c8a011ded4081f6e0ce095ba9f1ae08e6bd785cfea78906ea4fdaa371545e8"}
Nov 03 20:05:20.806881 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.806773    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93c8a011ded4081f6e0ce095ba9f1ae08e6bd785cfea78906ea4fdaa371545e8"
Nov 03 20:05:20.806881 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.806819    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"
Nov 03 20:05:20.872403 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.872313    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:20.872572 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.872423    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxjtj\" (UniqueName: \"kubernetes.io/projected/b9096d4f-4e10-4310-b688-6c7e64c371f7-kube-api-access-gxjtj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:20.872572 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.872485    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:20.873368 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.873338    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:20.885316 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.885262    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:20.886277 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.886248    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"]
Nov 03 20:05:20.888069 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.888027    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxjtj\" (UniqueName: \"kubernetes.io/projected/b9096d4f-4e10-4310-b688-6c7e64c371f7-kube-api-access-gxjtj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:20.989163 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:20.989133    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:21.098898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:21.098864    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:21.167008 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:21.166552    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:05:21.167008 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:21.166612    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" podUID="8d9e6662-96f2-42b0-853a-9649585967b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 20:05:21.406520 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:05:21.406487    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9096d4f_4e10_4310_b688_6c7e64c371f7.slice/crio-5acec54e65de5a2ea78ae44111472190076d0d9d2f62bdc3343a583fe1ca787e WatchSource:0}: Error finding container 5acec54e65de5a2ea78ae44111472190076d0d9d2f62bdc3343a583fe1ca787e: Status 404 returned error can't find the container with id 5acec54e65de5a2ea78ae44111472190076d0d9d2f62bdc3343a583fe1ca787e
Nov 03 20:05:21.407771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:21.407739    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"]
Nov 03 20:05:21.778197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:21.773801    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:21.833860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:21.833823    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748" event={"ID":"b9096d4f-4e10-4310-b688-6c7e64c371f7","Type":"ContainerStarted","Data":"5acec54e65de5a2ea78ae44111472190076d0d9d2f62bdc3343a583fe1ca787e"}
Nov 03 20:05:22.052456 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:22.051616    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:22.164569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:22.164521    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:05:22.164787 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:22.164600    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" podUID="8d9e6662-96f2-42b0-853a-9649585967b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 20:05:22.192460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:22.192424    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:22.378061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:22.377986    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:22.447151 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:22.447115    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:05:22.447596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:22.447563    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:05:22.711321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:22.711268    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:22.846589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:22.846449    4531 generic.go:356] "Generic (PLEG): container finished" podID="b9096d4f-4e10-4310-b688-6c7e64c371f7" containerID="8b0f454328db6a4f527a04ce6b0f808c7c312b49d5248ede15d9f14886558bad" exitCode=0
Nov 03 20:05:22.846589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:22.846531    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748" event={"ID":"b9096d4f-4e10-4310-b688-6c7e64c371f7","Type":"ContainerDied","Data":"8b0f454328db6a4f527a04ce6b0f808c7c312b49d5248ede15d9f14886558bad"}
Nov 03 20:05:23.203002 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:23.202963    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:23.203588 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:23.203564    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:23.204408 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:23.204385    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:05:23.771847 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:23.771812    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm" podUID="746b1f72-ee9f-4a64-bf2f-7db1ae805aef" containerName="payload-processing" containerID="cri-o://2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8" gracePeriod=25
Nov 03 20:05:23.788099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:23.788061    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4" podUID="3e71a759-fb64-494f-8e4d-6aa7a427f6e3" containerName="payload-pre-processing" containerID="cri-o://29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7" gracePeriod=25
Nov 03 20:05:23.914823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:23.914791    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:24.107810 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.107760    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:24.188440 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.188410    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 20:05:24.215483 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.215456    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 20:05:24.218482 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.218455    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-plugins-config-volume\") pod \"746b1f72-ee9f-4a64-bf2f-7db1ae805aef\" (UID: \"746b1f72-ee9f-4a64-bf2f-7db1ae805aef\") "
Nov 03 20:05:24.218645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.218626    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-xks-ca\") pod \"b9096d4f-4e10-4310-b688-6c7e64c371f7\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") "
Nov 03 20:05:24.218723 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.218708    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-openshift-ca\") pod \"b9096d4f-4e10-4310-b688-6c7e64c371f7\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") "
Nov 03 20:05:24.218886 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.218861    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-plugins-config-volume" pod "746b1f72-ee9f-4a64-bf2f-7db1ae805aef" (UID: "746b1f72-ee9f-4a64-bf2f-7db1ae805aef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:24.218951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.218925    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-kube-api-access-2m98t\" (UniqueName: \"kubernetes.io/projected/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-kube-api-access-2m98t\") pod \"746b1f72-ee9f-4a64-bf2f-7db1ae805aef\" (UID: \"746b1f72-ee9f-4a64-bf2f-7db1ae805aef\") "
Nov 03 20:05:24.219007 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.218973    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9096d4f-4e10-4310-b688-6c7e64c371f7-kube-api-access-gxjtj\" (UniqueName: \"kubernetes.io/projected/b9096d4f-4e10-4310-b688-6c7e64c371f7-kube-api-access-gxjtj\") pod \"b9096d4f-4e10-4310-b688-6c7e64c371f7\" (UID: \"b9096d4f-4e10-4310-b688-6c7e64c371f7\") "
Nov 03 20:05:24.219099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.219072    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-openshift-ca" pod "b9096d4f-4e10-4310-b688-6c7e64c371f7" (UID: "b9096d4f-4e10-4310-b688-6c7e64c371f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:24.219675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.219649    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:24.219752 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.219682    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:24.221787 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.221765    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-xks-ca" pod "b9096d4f-4e10-4310-b688-6c7e64c371f7" (UID: "b9096d4f-4e10-4310-b688-6c7e64c371f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:24.221963 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.221942    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-kube-api-access-2m98t" pod "746b1f72-ee9f-4a64-bf2f-7db1ae805aef" (UID: "746b1f72-ee9f-4a64-bf2f-7db1ae805aef"). InnerVolumeSpecName "kube-api-access-2m98t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:24.222070 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.222015    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9096d4f-4e10-4310-b688-6c7e64c371f7-kube-api-access-gxjtj" pod "b9096d4f-4e10-4310-b688-6c7e64c371f7" (UID: "b9096d4f-4e10-4310-b688-6c7e64c371f7"). InnerVolumeSpecName "kube-api-access-gxjtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:24.320397 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.320312    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-kube-api-access-qcm9x\" (UniqueName: \"kubernetes.io/projected/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-kube-api-access-qcm9x\") pod \"3e71a759-fb64-494f-8e4d-6aa7a427f6e3\" (UID: \"3e71a759-fb64-494f-8e4d-6aa7a427f6e3\") "
Nov 03 20:05:24.320397 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.320357    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-plugins-config-volume\") pod \"3e71a759-fb64-494f-8e4d-6aa7a427f6e3\" (UID: \"3e71a759-fb64-494f-8e4d-6aa7a427f6e3\") "
Nov 03 20:05:24.320782 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.320756    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-plugins-config-volume" pod "3e71a759-fb64-494f-8e4d-6aa7a427f6e3" (UID: "3e71a759-fb64-494f-8e4d-6aa7a427f6e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:24.320782 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.320775    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m98t\" (UniqueName: \"kubernetes.io/projected/746b1f72-ee9f-4a64-bf2f-7db1ae805aef-kube-api-access-2m98t\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:24.320901 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.320790    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxjtj\" (UniqueName: \"kubernetes.io/projected/b9096d4f-4e10-4310-b688-6c7e64c371f7-kube-api-access-gxjtj\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:24.320901 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.320800    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9096d4f-4e10-4310-b688-6c7e64c371f7-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:24.322638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.322608    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-kube-api-access-qcm9x" pod "3e71a759-fb64-494f-8e4d-6aa7a427f6e3" (UID: "3e71a759-fb64-494f-8e4d-6aa7a427f6e3"). InnerVolumeSpecName "kube-api-access-qcm9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:24.422303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.422262    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcm9x\" (UniqueName: \"kubernetes.io/projected/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-kube-api-access-qcm9x\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:24.422303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.422297    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e71a759-fb64-494f-8e4d-6aa7a427f6e3-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:24.676203 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.669418    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:24.865357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.865316    4531 generic.go:356] "Generic (PLEG): container finished" podID="3e71a759-fb64-494f-8e4d-6aa7a427f6e3" containerID="29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7" exitCode=0
Nov 03 20:05:24.865357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.865357    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"
Nov 03 20:05:24.865621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.865402    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4" event={"ID":"3e71a759-fb64-494f-8e4d-6aa7a427f6e3","Type":"ContainerDied","Data":"29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7"}
Nov 03 20:05:24.865621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.865451    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4" event={"ID":"3e71a759-fb64-494f-8e4d-6aa7a427f6e3","Type":"ContainerDied","Data":"fd3c9708a21210d962c4729de76ae33f89fe79d7d46c97f9e0e732eea37ba68d"}
Nov 03 20:05:24.865621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.865478    4531 scope.go:122] "RemoveContainer" containerID="29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7"
Nov 03 20:05:24.880548 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.880515    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748" event={"ID":"b9096d4f-4e10-4310-b688-6c7e64c371f7","Type":"ContainerDied","Data":"5acec54e65de5a2ea78ae44111472190076d0d9d2f62bdc3343a583fe1ca787e"}
Nov 03 20:05:24.880674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.880581    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5acec54e65de5a2ea78ae44111472190076d0d9d2f62bdc3343a583fe1ca787e"
Nov 03 20:05:24.880674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.880649    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"
Nov 03 20:05:24.882698 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.882671    4531 generic.go:356] "Generic (PLEG): container finished" podID="746b1f72-ee9f-4a64-bf2f-7db1ae805aef" containerID="2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8" exitCode=0
Nov 03 20:05:24.882828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.882707    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm" event={"ID":"746b1f72-ee9f-4a64-bf2f-7db1ae805aef","Type":"ContainerDied","Data":"2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8"}
Nov 03 20:05:24.882828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.882771    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm" event={"ID":"746b1f72-ee9f-4a64-bf2f-7db1ae805aef","Type":"ContainerDied","Data":"3b144541a973361fbc3e21e133b5c0c3bb38285a6812c50798cd125581d8c009"}
Nov 03 20:05:24.882828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.882717    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"
Nov 03 20:05:24.886318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.886295    4531 scope.go:122] "RemoveContainer" containerID="29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7"
Nov 03 20:05:24.886759 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:24.886733    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7\": container with ID starting with 29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7 not found: ID does not exist" containerID="29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7"
Nov 03 20:05:24.886856 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.886764    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7"} err="failed to get container status \"29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7\": rpc error: code = NotFound desc = could not find container \"29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7\": container with ID starting with 29363dddd1b607b0a3ad1ae66ed248c20a61725408bfa1af5e7984b9323ca0b7 not found: ID does not exist"
Nov 03 20:05:24.886856 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.886781    4531 scope.go:122] "RemoveContainer" containerID="2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8"
Nov 03 20:05:24.897242 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.897224    4531 scope.go:122] "RemoveContainer" containerID="2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8"
Nov 03 20:05:24.897534 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:24.897510    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8\": container with ID starting with 2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8 not found: ID does not exist" containerID="2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8"
Nov 03 20:05:24.897598 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.897542    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8"} err="failed to get container status \"2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8\": rpc error: code = NotFound desc = could not find container \"2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8\": container with ID starting with 2747cb1b7b2b29fdabce9573ab320d738310d4503403f399735c0d9383e372b8 not found: ID does not exist"
Nov 03 20:05:24.900749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.900727    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"]
Nov 03 20:05:24.905339 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.905316    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2d78ea-6f997454c9-j2kr4"]
Nov 03 20:05:24.915743 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.915719    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"]
Nov 03 20:05:24.919887 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:24.919863    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2d78ea-7f5d4c49b7-xgxbm"]
Nov 03 20:05:25.310466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.310359    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:25.454401 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.454376    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 20:05:25.540657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.540627    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-certs\") pod \"b9597b9a-67e3-492c-afb7-dc2e544b156d\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") "
Nov 03 20:05:25.540857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.540726    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-socket\") pod \"b9597b9a-67e3-492c-afb7-dc2e544b156d\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") "
Nov 03 20:05:25.540857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.540762    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-token\" (UniqueName: \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-token\") pod \"b9597b9a-67e3-492c-afb7-dc2e544b156d\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") "
Nov 03 20:05:25.540857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.540830    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-envoy\") pod \"b9597b9a-67e3-492c-afb7-dc2e544b156d\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") "
Nov 03 20:05:25.541030 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.540907    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-certs" pod "b9597b9a-67e3-492c-afb7-dc2e544b156d" (UID: "b9597b9a-67e3-492c-afb7-dc2e544b156d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:25.541030 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.540935    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-data\") pod \"b9597b9a-67e3-492c-afb7-dc2e544b156d\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") "
Nov 03 20:05:25.541138 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.541049    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-socket" pod "b9597b9a-67e3-492c-afb7-dc2e544b156d" (UID: "b9597b9a-67e3-492c-afb7-dc2e544b156d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:25.541138 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.541097    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9597b9a-67e3-492c-afb7-dc2e544b156d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9597b9a-67e3-492c-afb7-dc2e544b156d-istiod-ca-cert\") pod \"b9597b9a-67e3-492c-afb7-dc2e544b156d\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") "
Nov 03 20:05:25.541278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.541140    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-data" pod "b9597b9a-67e3-492c-afb7-dc2e544b156d" (UID: "b9597b9a-67e3-492c-afb7-dc2e544b156d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:25.541278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.541155    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-podinfo\") pod \"b9597b9a-67e3-492c-afb7-dc2e544b156d\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") "
Nov 03 20:05:25.541383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.541296    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-kube-api-access-tqrrt\" (UniqueName: \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-kube-api-access-tqrrt\") pod \"b9597b9a-67e3-492c-afb7-dc2e544b156d\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") "
Nov 03 20:05:25.541472 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.541378    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-credential-socket\") pod \"b9597b9a-67e3-492c-afb7-dc2e544b156d\" (UID: \"b9597b9a-67e3-492c-afb7-dc2e544b156d\") "
Nov 03 20:05:25.541472 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.541415    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9597b9a-67e3-492c-afb7-dc2e544b156d-istiod-ca-cert" pod "b9597b9a-67e3-492c-afb7-dc2e544b156d" (UID: "b9597b9a-67e3-492c-afb7-dc2e544b156d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:25.541704 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.541681    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-credential-socket" pod "b9597b9a-67e3-492c-afb7-dc2e544b156d" (UID: "b9597b9a-67e3-492c-afb7-dc2e544b156d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:25.542194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.542152    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9597b9a-67e3-492c-afb7-dc2e544b156d-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:25.542250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.542219    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:25.542250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.542235    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:25.542324 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.542251    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:25.542324 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.542267    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:25.543556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.543529    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-kube-api-access-tqrrt" pod "b9597b9a-67e3-492c-afb7-dc2e544b156d" (UID: "b9597b9a-67e3-492c-afb7-dc2e544b156d"). InnerVolumeSpecName "kube-api-access-tqrrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:25.543679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.543569    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-token" pod "b9597b9a-67e3-492c-afb7-dc2e544b156d" (UID: "b9597b9a-67e3-492c-afb7-dc2e544b156d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:25.543679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.543584    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-podinfo" pod "b9597b9a-67e3-492c-afb7-dc2e544b156d" (UID: "b9597b9a-67e3-492c-afb7-dc2e544b156d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:05:25.543983 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.543964    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-envoy" pod "b9597b9a-67e3-492c-afb7-dc2e544b156d" (UID: "b9597b9a-67e3-492c-afb7-dc2e544b156d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:25.643232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.643162    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:25.643232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.643226    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqrrt\" (UniqueName: \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-kube-api-access-tqrrt\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:25.643232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.643237    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:25.643489 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.643246    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9597b9a-67e3-492c-afb7-dc2e544b156d-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:25.815918 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.815780    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:25.892733 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.892694    4531 generic.go:356] "Generic (PLEG): container finished" podID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerID="a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2" exitCode=0
Nov 03 20:05:25.892933 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.892752    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"
Nov 03 20:05:25.892933 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.892768    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" event={"ID":"b9597b9a-67e3-492c-afb7-dc2e544b156d","Type":"ContainerDied","Data":"a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2"}
Nov 03 20:05:25.892933 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.892875    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm" event={"ID":"b9597b9a-67e3-492c-afb7-dc2e544b156d","Type":"ContainerDied","Data":"13b55486520515aa7727c1bfdc58539f52257cbf692ad4dc05328ac92cc18471"}
Nov 03 20:05:25.892933 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.892912    4531 scope.go:122] "RemoveContainer" containerID="a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2"
Nov 03 20:05:25.906081 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.906061    4531 scope.go:122] "RemoveContainer" containerID="246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660"
Nov 03 20:05:25.917288 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.914750    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"]
Nov 03 20:05:25.917827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.917806    4531 scope.go:122] "RemoveContainer" containerID="a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2"
Nov 03 20:05:25.918229 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:25.918167    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2\": container with ID starting with a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2 not found: ID does not exist" containerID="a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2"
Nov 03 20:05:25.918322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.918241    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2"} err="failed to get container status \"a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2\": rpc error: code = NotFound desc = could not find container \"a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2\": container with ID starting with a09374dcfa60fb682dbbf8b688ed9a01cbded69fb7fb0514d711da3b76f080c2 not found: ID does not exist"
Nov 03 20:05:25.918322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.918294    4531 scope.go:122] "RemoveContainer" containerID="246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660"
Nov 03 20:05:25.918784 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:25.918627    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660\": container with ID starting with 246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660 not found: ID does not exist" containerID="246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660"
Nov 03 20:05:25.918784 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.918660    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660"} err="failed to get container status \"246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660\": rpc error: code = NotFound desc = could not find container \"246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660\": container with ID starting with 246553624b44b2596eb0b71dad5b69032f6074a39c98821bd0131f98e5155660 not found: ID does not exist"
Nov 03 20:05:25.919753 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.919730    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2d78ea-fnhqs"]
Nov 03 20:05:25.923720 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.923699    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"]
Nov 03 20:05:25.927335 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.927316    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fa9526-openshift-default-677bd569b5-bs2wm"]
Nov 03 20:05:25.951098 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:25.951067    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:26.107248 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:26.100919    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:26.416458 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:26.416400    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:26.608265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:26.608094    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:26.872702 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:26.872597    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e71a759-fb64-494f-8e4d-6aa7a427f6e3" path="/var/lib/kubelet/pods/3e71a759-fb64-494f-8e4d-6aa7a427f6e3/volumes"
Nov 03 20:05:26.873322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:26.873284    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746b1f72-ee9f-4a64-bf2f-7db1ae805aef" path="/var/lib/kubelet/pods/746b1f72-ee9f-4a64-bf2f-7db1ae805aef/volumes"
Nov 03 20:05:26.873768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:26.873741    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca5dad5-b21a-4e0d-854d-d647e27d6775" path="/var/lib/kubelet/pods/9ca5dad5-b21a-4e0d-854d-d647e27d6775/volumes"
Nov 03 20:05:26.874738 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:26.874717    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" path="/var/lib/kubelet/pods/b9597b9a-67e3-492c-afb7-dc2e544b156d/volumes"
Nov 03 20:05:27.244200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:27.242301    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:27.913822 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:27.913789    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:28.556509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:28.553998    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:29.228197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.226862    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:29.653698 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.653655    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"]
Nov 03 20:05:29.654490 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654465    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy"
Nov 03 20:05:29.654490 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654490    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy"
Nov 03 20:05:29.654689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654516    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3e71a759-fb64-494f-8e4d-6aa7a427f6e3" containerName="payload-pre-processing"
Nov 03 20:05:29.654689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654523    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e71a759-fb64-494f-8e4d-6aa7a427f6e3" containerName="payload-pre-processing"
Nov 03 20:05:29.654689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654549    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b9096d4f-4e10-4310-b688-6c7e64c371f7" containerName="revoke-keys"
Nov 03 20:05:29.654689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654557    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9096d4f-4e10-4310-b688-6c7e64c371f7" containerName="revoke-keys"
Nov 03 20:05:29.654689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654577    4531 cpu_manager.go:414] "containerMap: removing container" podUID="746b1f72-ee9f-4a64-bf2f-7db1ae805aef" containerName="payload-processing"
Nov 03 20:05:29.654689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654585    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746b1f72-ee9f-4a64-bf2f-7db1ae805aef" containerName="payload-processing"
Nov 03 20:05:29.654995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654761    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy"
Nov 03 20:05:29.654995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654776    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e71a759-fb64-494f-8e4d-6aa7a427f6e3" containerName="payload-pre-processing"
Nov 03 20:05:29.654995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654790    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9096d4f-4e10-4310-b688-6c7e64c371f7" containerName="revoke-keys"
Nov 03 20:05:29.654995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654800    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="746b1f72-ee9f-4a64-bf2f-7db1ae805aef" containerName="payload-processing"
Nov 03 20:05:29.654995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.654814    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy"
Nov 03 20:05:29.660725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.660687    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:29.665128 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.665050    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"]
Nov 03 20:05:29.666845 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.666819    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:05:29.666845 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.666818    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:05:29.738776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.738740    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c3e245-8841-403a-a185-1de590d57058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442859bb-4nnxv\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:29.738931 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.738817    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c3e245-8841-403a-a185-1de590d57058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442859bb-4nnxv\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:29.739047 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.739034    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8j4j\" (UniqueName: \"kubernetes.io/projected/b0c3e245-8841-403a-a185-1de590d57058-kube-api-access-x8j4j\") pod \"maas-api-revoke-sub-442859bb-4nnxv\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:29.840142 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.840111    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c3e245-8841-403a-a185-1de590d57058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442859bb-4nnxv\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:29.840360 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.840317    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8j4j\" (UniqueName: \"kubernetes.io/projected/b0c3e245-8841-403a-a185-1de590d57058-kube-api-access-x8j4j\") pod \"maas-api-revoke-sub-442859bb-4nnxv\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:29.840432 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.840389    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c3e245-8841-403a-a185-1de590d57058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442859bb-4nnxv\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:29.840769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.840742    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c3e245-8841-403a-a185-1de590d57058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442859bb-4nnxv\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:29.842827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.842791    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c3e245-8841-403a-a185-1de590d57058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442859bb-4nnxv\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:29.847100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.847079    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8j4j\" (UniqueName: \"kubernetes.io/projected/b0c3e245-8841-403a-a185-1de590d57058-kube-api-access-x8j4j\") pod \"maas-api-revoke-sub-442859bb-4nnxv\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:29.959260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.951003    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:29.985207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:29.984430    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:30.283003 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:30.282971    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"]
Nov 03 20:05:30.283509 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:05:30.283384    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c3e245_8841_403a_a185_1de590d57058.slice/crio-a8ea99f9ed936e9258b1f6b500197daa3db2b367dc16817c89ca0af3671ba080 WatchSource:0}: Error finding container a8ea99f9ed936e9258b1f6b500197daa3db2b367dc16817c89ca0af3671ba080: Status 404 returned error can't find the container with id a8ea99f9ed936e9258b1f6b500197daa3db2b367dc16817c89ca0af3671ba080
Nov 03 20:05:30.657195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:30.647115    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:30.920602 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:30.920573    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:30.939554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:30.939519    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv" event={"ID":"b0c3e245-8841-403a-a185-1de590d57058","Type":"ContainerStarted","Data":"f82c622a18f371fff5d3e8c6c77fe29c569481d93db884d42c37ea2d463a138b"}
Nov 03 20:05:30.939761 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:30.939574    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv" event={"ID":"b0c3e245-8841-403a-a185-1de590d57058","Type":"ContainerStarted","Data":"a8ea99f9ed936e9258b1f6b500197daa3db2b367dc16817c89ca0af3671ba080"}
Nov 03 20:05:30.954606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:30.954549    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv" podStartSLOduration=1.954535456 podStartE2EDuration="1.954535456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:30.952214642 +0000 UTC m=+2044.222337937" watchObservedRunningTime="2026-11-03 20:05:30.954535456 +0000 UTC m=+2044.224658750"
Nov 03 20:05:31.053865 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:31.053827    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:31.427614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:31.427560    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:31.950455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:31.950421    4531 generic.go:356] "Generic (PLEG): container finished" podID="b0c3e245-8841-403a-a185-1de590d57058" containerID="f82c622a18f371fff5d3e8c6c77fe29c569481d93db884d42c37ea2d463a138b" exitCode=0
Nov 03 20:05:31.950642 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:31.950465    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv" event={"ID":"b0c3e245-8841-403a-a185-1de590d57058","Type":"ContainerDied","Data":"f82c622a18f371fff5d3e8c6c77fe29c569481d93db884d42c37ea2d463a138b"}
Nov 03 20:05:32.483836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:32.483792    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:33.284060 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.283817    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:33.313986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.313956    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:33.390745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.390703    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0c3e245-8841-403a-a185-1de590d57058-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c3e245-8841-403a-a185-1de590d57058-maas-api-xks-ca\") pod \"b0c3e245-8841-403a-a185-1de590d57058\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") "
Nov 03 20:05:33.390951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.390766    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c3e245-8841-403a-a185-1de590d57058-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c3e245-8841-403a-a185-1de590d57058-maas-api-openshift-ca\") pod \"b0c3e245-8841-403a-a185-1de590d57058\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") "
Nov 03 20:05:33.391124 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.391101    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c3e245-8841-403a-a185-1de590d57058-kube-api-access-x8j4j\" (UniqueName: \"kubernetes.io/projected/b0c3e245-8841-403a-a185-1de590d57058-kube-api-access-x8j4j\") pod \"b0c3e245-8841-403a-a185-1de590d57058\" (UID: \"b0c3e245-8841-403a-a185-1de590d57058\") "
Nov 03 20:05:33.393118 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.393088    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c3e245-8841-403a-a185-1de590d57058-maas-api-openshift-ca" pod "b0c3e245-8841-403a-a185-1de590d57058" (UID: "b0c3e245-8841-403a-a185-1de590d57058"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:33.395385 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.395353    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c3e245-8841-403a-a185-1de590d57058-maas-api-xks-ca" pod "b0c3e245-8841-403a-a185-1de590d57058" (UID: "b0c3e245-8841-403a-a185-1de590d57058"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:33.397108 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.396975    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c3e245-8841-403a-a185-1de590d57058-kube-api-access-x8j4j" pod "b0c3e245-8841-403a-a185-1de590d57058" (UID: "b0c3e245-8841-403a-a185-1de590d57058"). InnerVolumeSpecName "kube-api-access-x8j4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:33.461210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.459917    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:33.498571 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.498529    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8j4j\" (UniqueName: \"kubernetes.io/projected/b0c3e245-8841-403a-a185-1de590d57058-kube-api-access-x8j4j\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:33.499290 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.499256    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c3e245-8841-403a-a185-1de590d57058-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:33.499478 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.499462    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c3e245-8841-403a-a185-1de590d57058-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:33.601644 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.601551    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:33.896705 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:33.896672    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:34.062922 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.062888    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:34.101196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.097803    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv" event={"ID":"b0c3e245-8841-403a-a185-1de590d57058","Type":"ContainerDied","Data":"a8ea99f9ed936e9258b1f6b500197daa3db2b367dc16817c89ca0af3671ba080"}
Nov 03 20:05:34.101196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.097879    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8ea99f9ed936e9258b1f6b500197daa3db2b367dc16817c89ca0af3671ba080"
Nov 03 20:05:34.101196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.097957    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"
Nov 03 20:05:34.227460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.227418    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:34.588726 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.588470    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:34.724922 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.724894    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:34.784932 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.784893    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"]
Nov 03 20:05:34.785618 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.785603    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b0c3e245-8841-403a-a185-1de590d57058" containerName="revoke-keys"
Nov 03 20:05:34.785689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.785619    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c3e245-8841-403a-a185-1de590d57058" containerName="revoke-keys"
Nov 03 20:05:34.785689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.785650    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy"
Nov 03 20:05:34.785689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.785654    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9597b9a-67e3-492c-afb7-dc2e544b156d" containerName="istio-proxy"
Nov 03 20:05:34.785791 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.785747    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c3e245-8841-403a-a185-1de590d57058" containerName="revoke-keys"
Nov 03 20:05:34.789668 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.789646    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:34.792672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.792651    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:05:34.792836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.792727    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:05:34.797307 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.797278    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"]
Nov 03 20:05:34.862379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.862265    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:34.862572 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.862417    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:34.862572 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.862464    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9ld\" (UniqueName: \"kubernetes.io/projected/29ceae64-9c23-4083-8d78-452dad1c3db0-kube-api-access-jf9ld\") pod \"maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:34.964005 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.963963    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:34.964243 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.964094    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:34.964243 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.964141    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9ld\" (UniqueName: \"kubernetes.io/projected/29ceae64-9c23-4083-8d78-452dad1c3db0-kube-api-access-jf9ld\") pod \"maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:34.965321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.965293    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:34.966886 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.966852    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:34.973775 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:34.973746    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9ld\" (UniqueName: \"kubernetes.io/projected/29ceae64-9c23-4083-8d78-452dad1c3db0-kube-api-access-jf9ld\") pod \"maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:35.109935 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:35.109900    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:35.464225 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:05:35.461664    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ceae64_9c23_4083_8d78_452dad1c3db0.slice/crio-2a3cd83850d52233866fdd85cce76ed40d73f7fb80933c0ff346e0faa2d8cc2d WatchSource:0}: Error finding container 2a3cd83850d52233866fdd85cce76ed40d73f7fb80933c0ff346e0faa2d8cc2d: Status 404 returned error can't find the container with id 2a3cd83850d52233866fdd85cce76ed40d73f7fb80933c0ff346e0faa2d8cc2d
Nov 03 20:05:35.466809 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:35.466751    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"]
Nov 03 20:05:35.715795 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:35.715751    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:35.909005 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:35.908966    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main" containerID="cri-o://2fd6a943663d357d6788d3f1af7af3531e66d9477f007e5c13cbd6a2b324a3d4" gracePeriod=45
Nov 03 20:05:36.122148 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.122085    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq" event={"ID":"29ceae64-9c23-4083-8d78-452dad1c3db0","Type":"ContainerStarted","Data":"06b2824314de2741d868d0fa5b55cff24ffef052971970e779734b917e871af1"}
Nov 03 20:05:36.122329 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.122208    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq" event={"ID":"29ceae64-9c23-4083-8d78-452dad1c3db0","Type":"ContainerStarted","Data":"2a3cd83850d52233866fdd85cce76ed40d73f7fb80933c0ff346e0faa2d8cc2d"}
Nov 03 20:05:36.125136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.125108    4531 generic.go:356] "Generic (PLEG): container finished" podID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerID="2fd6a943663d357d6788d3f1af7af3531e66d9477f007e5c13cbd6a2b324a3d4" exitCode=0
Nov 03 20:05:36.125263 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.125244    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" event={"ID":"4bca7c22-ce45-445e-ba29-c522e440abe9","Type":"ContainerDied","Data":"2fd6a943663d357d6788d3f1af7af3531e66d9477f007e5c13cbd6a2b324a3d4"}
Nov 03 20:05:36.268093 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.268028    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 20:05:36.282949 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.282895    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq" podStartSLOduration=2.282879266 podStartE2EDuration="2.282879266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:36.166459701 +0000 UTC m=+2049.436583005" watchObservedRunningTime="2026-11-03 20:05:36.282879266 +0000 UTC m=+2049.553002559"
Nov 03 20:05:36.291424 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.291392    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bca7c22-ce45-445e-ba29-c522e440abe9-kube-api-access-lw558\" (UniqueName: \"kubernetes.io/projected/4bca7c22-ce45-445e-ba29-c522e440abe9-kube-api-access-lw558\") pod \"4bca7c22-ce45-445e-ba29-c522e440abe9\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") "
Nov 03 20:05:36.291613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.291465    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-tmp-dir\") pod \"4bca7c22-ce45-445e-ba29-c522e440abe9\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") "
Nov 03 20:05:36.291613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.291499    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-dshm\") pod \"4bca7c22-ce45-445e-ba29-c522e440abe9\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") "
Nov 03 20:05:36.291613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.291594    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bca7c22-ce45-445e-ba29-c522e440abe9-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bca7c22-ce45-445e-ba29-c522e440abe9-tls-certs\") pod \"4bca7c22-ce45-445e-ba29-c522e440abe9\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") "
Nov 03 20:05:36.291767 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.291689    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-home\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-home\") pod \"4bca7c22-ce45-445e-ba29-c522e440abe9\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") "
Nov 03 20:05:36.291872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.291853    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-model-cache\") pod \"4bca7c22-ce45-445e-ba29-c522e440abe9\" (UID: \"4bca7c22-ce45-445e-ba29-c522e440abe9\") "
Nov 03 20:05:36.292398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.292369    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-tmp-dir" pod "4bca7c22-ce45-445e-ba29-c522e440abe9" (UID: "4bca7c22-ce45-445e-ba29-c522e440abe9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:36.292588 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.292559    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:36.292833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.292810    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-home" pod "4bca7c22-ce45-445e-ba29-c522e440abe9" (UID: "4bca7c22-ce45-445e-ba29-c522e440abe9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:36.292914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.292871    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-model-cache" pod "4bca7c22-ce45-445e-ba29-c522e440abe9" (UID: "4bca7c22-ce45-445e-ba29-c522e440abe9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:36.296787 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.296763    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-dshm" pod "4bca7c22-ce45-445e-ba29-c522e440abe9" (UID: "4bca7c22-ce45-445e-ba29-c522e440abe9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:36.298526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.298498    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bca7c22-ce45-445e-ba29-c522e440abe9-kube-api-access-lw558" pod "4bca7c22-ce45-445e-ba29-c522e440abe9" (UID: "4bca7c22-ce45-445e-ba29-c522e440abe9"). InnerVolumeSpecName "kube-api-access-lw558". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:36.299972 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.299948    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bca7c22-ce45-445e-ba29-c522e440abe9-tls-certs" pod "4bca7c22-ce45-445e-ba29-c522e440abe9" (UID: "4bca7c22-ce45-445e-ba29-c522e440abe9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:36.394347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.394310    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bca7c22-ce45-445e-ba29-c522e440abe9-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:36.394347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.394338    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:36.394347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.394349    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:36.394593 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.394359    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw558\" (UniqueName: \"kubernetes.io/projected/4bca7c22-ce45-445e-ba29-c522e440abe9-kube-api-access-lw558\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:36.394593 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.394369    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bca7c22-ce45-445e-ba29-c522e440abe9-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:36.657228 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:36.657067    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:37.134998 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:37.134914    4531 generic.go:356] "Generic (PLEG): container finished" podID="29ceae64-9c23-4083-8d78-452dad1c3db0" containerID="06b2824314de2741d868d0fa5b55cff24ffef052971970e779734b917e871af1" exitCode=0
Nov 03 20:05:37.135508 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:37.134994    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq" event={"ID":"29ceae64-9c23-4083-8d78-452dad1c3db0","Type":"ContainerDied","Data":"06b2824314de2741d868d0fa5b55cff24ffef052971970e779734b917e871af1"}
Nov 03 20:05:37.160197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:37.158604    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg" event={"ID":"4bca7c22-ce45-445e-ba29-c522e440abe9","Type":"ContainerDied","Data":"c0918ce1ac97d62d00a481e84c08cce233ce9287adbe1143a68a5869958fc1b5"}
Nov 03 20:05:37.160197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:37.158666    4531 scope.go:122] "RemoveContainer" containerID="2fd6a943663d357d6788d3f1af7af3531e66d9477f007e5c13cbd6a2b324a3d4"
Nov 03 20:05:37.160197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:37.158872    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"
Nov 03 20:05:37.188836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:37.188801    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"]
Nov 03 20:05:37.192672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:37.192651    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/simulator-w3-39043d-kserve-7df4d6c4c9-d2lxg"]
Nov 03 20:05:37.474248 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:37.466013    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:38.371464 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.318268    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:38.434813 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.434736    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:38.533234 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.533196    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-openshift-ca\") pod \"29ceae64-9c23-4083-8d78-452dad1c3db0\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") "
Nov 03 20:05:38.533436 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.533324    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-xks-ca\") pod \"29ceae64-9c23-4083-8d78-452dad1c3db0\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") "
Nov 03 20:05:38.533609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.533593    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ceae64-9c23-4083-8d78-452dad1c3db0-kube-api-access-jf9ld\" (UniqueName: \"kubernetes.io/projected/29ceae64-9c23-4083-8d78-452dad1c3db0-kube-api-access-jf9ld\") pod \"29ceae64-9c23-4083-8d78-452dad1c3db0\" (UID: \"29ceae64-9c23-4083-8d78-452dad1c3db0\") "
Nov 03 20:05:38.533675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.533595    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-openshift-ca" pod "29ceae64-9c23-4083-8d78-452dad1c3db0" (UID: "29ceae64-9c23-4083-8d78-452dad1c3db0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:38.534299 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.534271    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:38.542334 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.542281    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-xks-ca" pod "29ceae64-9c23-4083-8d78-452dad1c3db0" (UID: "29ceae64-9c23-4083-8d78-452dad1c3db0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:38.563821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.563780    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ceae64-9c23-4083-8d78-452dad1c3db0-kube-api-access-jf9ld" pod "29ceae64-9c23-4083-8d78-452dad1c3db0" (UID: "29ceae64-9c23-4083-8d78-452dad1c3db0"). InnerVolumeSpecName "kube-api-access-jf9ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:38.635587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.635531    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf9ld\" (UniqueName: \"kubernetes.io/projected/29ceae64-9c23-4083-8d78-452dad1c3db0-kube-api-access-jf9ld\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:38.635587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.635588    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ceae64-9c23-4083-8d78-452dad1c3db0-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:38.866182 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:38.866136    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" path="/var/lib/kubelet/pods/4bca7c22-ce45-445e-ba29-c522e440abe9/volumes"
Nov 03 20:05:39.137255 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:39.132824    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:39.189342 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:39.189289    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq" event={"ID":"29ceae64-9c23-4083-8d78-452dad1c3db0","Type":"ContainerDied","Data":"2a3cd83850d52233866fdd85cce76ed40d73f7fb80933c0ff346e0faa2d8cc2d"}
Nov 03 20:05:39.189606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:39.189589    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a3cd83850d52233866fdd85cce76ed40d73f7fb80933c0ff346e0faa2d8cc2d"
Nov 03 20:05:39.189800 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:39.189787    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"
Nov 03 20:05:39.601707 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:39.601624    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"]
Nov 03 20:05:39.727024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:39.726914    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:40.131133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:40.128703    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:40.800194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:40.790040    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:41.467196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:41.436940    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:42.023698 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:42.022469    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"]
Nov 03 20:05:42.026370 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:42.026339    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06962bc6-kb5rx"]
Nov 03 20:05:42.153195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:42.142903    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:42.593636 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:42.593596    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"]
Nov 03 20:05:42.594402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:42.593846    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw" podUID="8b03a364-09e1-4aa1-9644-eaff7261c5dd" containerName="maas-api" containerID="cri-o://37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901" gracePeriod=30
Nov 03 20:05:42.665042 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:42.665010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:42.829237 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:42.824938    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:42.991196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:42.954402    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc5d5e9-df4c-4644-b1a3-1577b98c9a80" path="/var/lib/kubelet/pods/acc5d5e9-df4c-4644-b1a3-1577b98c9a80/volumes"
Nov 03 20:05:43.015360 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.013233    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:43.040087 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.040051    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"]
Nov 03 20:05:43.044600 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.044564    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-10572d-ck748"]
Nov 03 20:05:43.131262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.127115    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 20:05:43.131262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.128693    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b03a364-09e1-4aa1-9644-eaff7261c5dd-kube-api-access-2cdcf\" (UniqueName: \"kubernetes.io/projected/8b03a364-09e1-4aa1-9644-eaff7261c5dd-kube-api-access-2cdcf\") pod \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") "
Nov 03 20:05:43.131262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.128787    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b03a364-09e1-4aa1-9644-eaff7261c5dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b03a364-09e1-4aa1-9644-eaff7261c5dd-openshift-service-ca\") pod \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") "
Nov 03 20:05:43.131262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.128874    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-maas-api-tls\") pod \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") "
Nov 03 20:05:43.131262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.128965    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-metrics-certs\") pod \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\" (UID: \"8b03a364-09e1-4aa1-9644-eaff7261c5dd\") "
Nov 03 20:05:43.142240 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.139629    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b03a364-09e1-4aa1-9644-eaff7261c5dd-openshift-service-ca" pod "8b03a364-09e1-4aa1-9644-eaff7261c5dd" (UID: "8b03a364-09e1-4aa1-9644-eaff7261c5dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:43.142240 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.142145    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b03a364-09e1-4aa1-9644-eaff7261c5dd-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:43.148691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.148197    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:43.153220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.153040    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-metrics-certs" pod "8b03a364-09e1-4aa1-9644-eaff7261c5dd" (UID: "8b03a364-09e1-4aa1-9644-eaff7261c5dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:43.153478 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.153457    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b03a364-09e1-4aa1-9644-eaff7261c5dd-kube-api-access-2cdcf" pod "8b03a364-09e1-4aa1-9644-eaff7261c5dd" (UID: "8b03a364-09e1-4aa1-9644-eaff7261c5dd"). InnerVolumeSpecName "kube-api-access-2cdcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:43.153628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.153588    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-maas-api-tls" pod "8b03a364-09e1-4aa1-9644-eaff7261c5dd" (UID: "8b03a364-09e1-4aa1-9644-eaff7261c5dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:43.225737 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.225701    4531 generic.go:356] "Generic (PLEG): container finished" podID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerID="58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982" exitCode=0
Nov 03 20:05:43.225950 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.225775    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" event={"ID":"7e747c83-6f8d-47ac-8023-d86ce46bcf94","Type":"ContainerDied","Data":"58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982"}
Nov 03 20:05:43.226436 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.226405    4531 scope.go:122] "RemoveContainer" containerID="58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982"
Nov 03 20:05:43.227558 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.227538    4531 generic.go:356] "Generic (PLEG): container finished" podID="8b03a364-09e1-4aa1-9644-eaff7261c5dd" containerID="37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901" exitCode=0
Nov 03 20:05:43.227662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.227630    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"
Nov 03 20:05:43.227727 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.227675    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw" event={"ID":"8b03a364-09e1-4aa1-9644-eaff7261c5dd","Type":"ContainerDied","Data":"37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901"}
Nov 03 20:05:43.227777 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.227741    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw" event={"ID":"8b03a364-09e1-4aa1-9644-eaff7261c5dd","Type":"ContainerDied","Data":"c09087a3db901ab6a8d490e3748c7aa11029a5a96b9108e01409b193690babcf"}
Nov 03 20:05:43.227777 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.227767    4531 scope.go:122] "RemoveContainer" containerID="37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901"
Nov 03 20:05:43.228423 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.228369    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:43.228544 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.228508    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:43.228615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.228558    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:43.243293 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.243267    4531 scope.go:122] "RemoveContainer" containerID="37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901"
Nov 03 20:05:43.243707 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:43.243678    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901\": container with ID starting with 37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901 not found: ID does not exist" containerID="37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901"
Nov 03 20:05:43.243858 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.243718    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901"} err="failed to get container status \"37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901\": rpc error: code = NotFound desc = could not find container \"37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901\": container with ID starting with 37de1b6fb47a9fdf00bd02e3eb078df39c005b83b53b917f4d69a1a9615f7901 not found: ID does not exist"
Nov 03 20:05:43.245132 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.245095    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cdcf\" (UniqueName: \"kubernetes.io/projected/8b03a364-09e1-4aa1-9644-eaff7261c5dd-kube-api-access-2cdcf\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:43.245132 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.245121    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:43.245132 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.245132    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b03a364-09e1-4aa1-9644-eaff7261c5dd-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:43.267328 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.267291    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"]
Nov 03 20:05:43.281019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.280985    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-10572d-79bfb459dd-kvfbw"]
Nov 03 20:05:43.462937 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.462850    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:43.749362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.749272    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:43.940197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:43.934367    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:44.093056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:44.092157    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:44.251206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:44.241122    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:44.300228 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:44.300160    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" event={"ID":"7e747c83-6f8d-47ac-8023-d86ce46bcf94","Type":"ContainerStarted","Data":"7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19"}
Nov 03 20:05:44.590526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:44.590492    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:44.722029 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:44.721995    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:05:44.723630 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:44.723602    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 20:05:44.723770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:44.723652    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 20:05:44.869402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:44.869324    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b03a364-09e1-4aa1-9644-eaff7261c5dd" path="/var/lib/kubelet/pods/8b03a364-09e1-4aa1-9644-eaff7261c5dd/volumes"
Nov 03 20:05:44.869873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:44.869852    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9096d4f-4e10-4310-b688-6c7e64c371f7" path="/var/lib/kubelet/pods/b9096d4f-4e10-4310-b688-6c7e64c371f7/volumes"
Nov 03 20:05:44.967487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:44.967451    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"]
Nov 03 20:05:45.063823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:45.063782    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:45.217607 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:45.217566    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:45.380208 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:45.379756    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:45.721347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:45.721308    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 20:05:45.721541 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:45.721370    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 20:05:45.774254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:45.774221    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:45.974250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:45.969159    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:46.084051 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:46.083977    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"]
Nov 03 20:05:46.095582 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:46.095543    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39043d-n85zv"]
Nov 03 20:05:46.137657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:46.137622    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:46.289782 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:46.289645    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:46.571486 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:46.571398    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:46.721676 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:46.721639    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 20:05:46.721898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:46.721699    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 20:05:46.875406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:46.875318    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24716c14-b72e-4d1e-b51d-34b1f587bd92" path="/var/lib/kubelet/pods/24716c14-b72e-4d1e-b51d-34b1f587bd92/volumes"
Nov 03 20:05:47.088757 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:47.088716    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:05:47.184333 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:47.184266    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:47.722418 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:47.722374    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 20:05:47.722871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:47.722434    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 20:05:48.170912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:48.170873    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:48.721498 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:48.721461    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 20:05:48.721692 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:48.721521    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 20:05:48.990251 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:48.979680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:49.673411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:49.673370    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:49.721578 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:49.721542    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 20:05:49.721834 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:49.721804    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 20:05:49.852287 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:49.852164    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:49.968009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:49.967905    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4" podUID="b511ca1a-779c-45e1-b233-55b34ba02945" containerName="payload-pre-processing" containerID="cri-o://a5e566870b104275c241ab2d21a85e6c9ecb3d13c06abd841be76dc6222af1a9" gracePeriod=25
Nov 03 20:05:49.988591 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:49.988562    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:50.093138 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.093108    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"]
Nov 03 20:05:50.093647 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.093383    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy" containerID="cri-o://4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651" gracePeriod=30
Nov 03 20:05:50.386747 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.384443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:50.417017 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.416987    4531 generic.go:356] "Generic (PLEG): container finished" podID="b511ca1a-779c-45e1-b233-55b34ba02945" containerID="a5e566870b104275c241ab2d21a85e6c9ecb3d13c06abd841be76dc6222af1a9" exitCode=0
Nov 03 20:05:50.417214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.417057    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4" event={"ID":"b511ca1a-779c-45e1-b233-55b34ba02945","Type":"ContainerDied","Data":"a5e566870b104275c241ab2d21a85e6c9ecb3d13c06abd841be76dc6222af1a9"}
Nov 03 20:05:50.417214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.417116    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4" event={"ID":"b511ca1a-779c-45e1-b233-55b34ba02945","Type":"ContainerDied","Data":"5faa00ec42cb25a9ca95524179e56000b748121fbccbffc88b2048f972760996"}
Nov 03 20:05:50.417214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.417133    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5faa00ec42cb25a9ca95524179e56000b748121fbccbffc88b2048f972760996"
Nov 03 20:05:50.418374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.418353    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 20:05:50.456570 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.456540    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b511ca1a-779c-45e1-b233-55b34ba02945-kube-api-access-bmw7v\" (UniqueName: \"kubernetes.io/projected/b511ca1a-779c-45e1-b233-55b34ba02945-kube-api-access-bmw7v\") pod \"b511ca1a-779c-45e1-b233-55b34ba02945\" (UID: \"b511ca1a-779c-45e1-b233-55b34ba02945\") "
Nov 03 20:05:50.456783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.456766    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b511ca1a-779c-45e1-b233-55b34ba02945-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b511ca1a-779c-45e1-b233-55b34ba02945-plugins-config-volume\") pod \"b511ca1a-779c-45e1-b233-55b34ba02945\" (UID: \"b511ca1a-779c-45e1-b233-55b34ba02945\") "
Nov 03 20:05:50.457232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.457189    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b511ca1a-779c-45e1-b233-55b34ba02945-plugins-config-volume" pod "b511ca1a-779c-45e1-b233-55b34ba02945" (UID: "b511ca1a-779c-45e1-b233-55b34ba02945"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:50.458917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.458891    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b511ca1a-779c-45e1-b233-55b34ba02945-kube-api-access-bmw7v" pod "b511ca1a-779c-45e1-b233-55b34ba02945" (UID: "b511ca1a-779c-45e1-b233-55b34ba02945"). InnerVolumeSpecName "kube-api-access-bmw7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:50.559112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.559076    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b511ca1a-779c-45e1-b233-55b34ba02945-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:50.559112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.559102    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmw7v\" (UniqueName: \"kubernetes.io/projected/b511ca1a-779c-45e1-b233-55b34ba02945-kube-api-access-bmw7v\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:50.575130 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:50.575092    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:05:50.655799 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.655736    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.762947    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"]
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.765898    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main"
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.765924    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main"
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.765957    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8b03a364-09e1-4aa1-9644-eaff7261c5dd" containerName="maas-api"
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.765965    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b03a364-09e1-4aa1-9644-eaff7261c5dd" containerName="maas-api"
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.765989    4531 cpu_manager.go:414] "containerMap: removing container" podUID="29ceae64-9c23-4083-8d78-452dad1c3db0" containerName="revoke-keys"
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.765997    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ceae64-9c23-4083-8d78-452dad1c3db0" containerName="revoke-keys"
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.766044    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b511ca1a-779c-45e1-b233-55b34ba02945" containerName="payload-pre-processing"
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.766052    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b511ca1a-779c-45e1-b233-55b34ba02945" containerName="payload-pre-processing"
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.766268    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b511ca1a-779c-45e1-b233-55b34ba02945" containerName="payload-pre-processing"
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.766287    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bca7c22-ce45-445e-ba29-c522e440abe9" containerName="main"
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.766304    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b03a364-09e1-4aa1-9644-eaff7261c5dd" containerName="maas-api"
Nov 03 20:05:50.771673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.766318    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ceae64-9c23-4083-8d78-452dad1c3db0" containerName="revoke-keys"
Nov 03 20:05:50.774495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.774457    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:50.774724 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.774690    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:05:50.778241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.776921    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:05:50.778241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.777021    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:05:50.778882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.778858    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:05:50.790198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.784759    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"]
Nov 03 20:05:50.819260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.819162    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:50.871260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.870011    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:50.871260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.870076    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz6vf\" (UniqueName: \"kubernetes.io/projected/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-kube-api-access-qz6vf\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:50.871260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.870307    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:50.972160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.972073    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:50.972160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.972133    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz6vf\" (UniqueName: \"kubernetes.io/projected/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-kube-api-access-qz6vf\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:50.972671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.972644    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:50.973089 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.973068    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:50.975523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.975502    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:50.979006 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:50.978971    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz6vf\" (UniqueName: \"kubernetes.io/projected/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-kube-api-access-qz6vf\") pod \"maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:51.104747 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:51.104708    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:51.115426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:51.115396    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:51.303244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:51.303009    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:51.425478 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:51.425444    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"
Nov 03 20:05:51.448219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:51.446574    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"]
Nov 03 20:05:51.450674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:51.450643    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a35658-5fdcfb57f5-5wrx4"]
Nov 03 20:05:51.650882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:51.631226    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"]
Nov 03 20:05:51.659662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:51.659631    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"]
Nov 03 20:05:51.659826 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:51.659806    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" podUID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" containerName="istio-proxy" containerID="cri-o://7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb" gracePeriod=30
Nov 03 20:05:51.734216 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:51.716443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:52.028264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:52.028221    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:52.436319 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:52.436219    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc" event={"ID":"cfa1c1ee-7221-4575-b8c8-dd10e30aef62","Type":"ContainerStarted","Data":"26e15ca5c9aecabd4a1aff4da8ddcd3681b522af59407f06624eff5b06cc9479"}
Nov 03 20:05:52.436319 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:52.436287    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc" event={"ID":"cfa1c1ee-7221-4575-b8c8-dd10e30aef62","Type":"ContainerStarted","Data":"31302fbc0f4b2e917ed94d494ca0ca0943f0b126e971d9afd863d359e660c0f5"}
Nov 03 20:05:52.450373 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:52.450076    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc" podStartSLOduration=2.450051099 podStartE2EDuration="2.450051099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:52.449465677 +0000 UTC m=+2065.719588966" watchObservedRunningTime="2026-11-03 20:05:52.450051099 +0000 UTC m=+2065.720174393"
Nov 03 20:05:52.716218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:52.713185    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:52.851740 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:52.851702    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:52.859449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:52.859422    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b511ca1a-779c-45e1-b233-55b34ba02945" path="/var/lib/kubelet/pods/b511ca1a-779c-45e1-b233-55b34ba02945/volumes"
Nov 03 20:05:52.977271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:52.969229    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"]
Nov 03 20:05:52.977271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:52.969466    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy" containerID="cri-o://82fefa89998d203254f5768fff02651ed935b615ecb2644d6ef2086f43341258" gracePeriod=30
Nov 03 20:05:53.003639 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:53.003600    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"]
Nov 03 20:05:53.004125 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:53.003875    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy" containerID="cri-o://6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83" gracePeriod=30
Nov 03 20:05:53.012694 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:53.011533    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:53.158211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:53.146912    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:53.446926 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:53.446888    4531 generic.go:356] "Generic (PLEG): container finished" podID="cfa1c1ee-7221-4575-b8c8-dd10e30aef62" containerID="26e15ca5c9aecabd4a1aff4da8ddcd3681b522af59407f06624eff5b06cc9479" exitCode=0
Nov 03 20:05:53.447405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:53.446970    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc" event={"ID":"cfa1c1ee-7221-4575-b8c8-dd10e30aef62","Type":"ContainerDied","Data":"26e15ca5c9aecabd4a1aff4da8ddcd3681b522af59407f06624eff5b06cc9479"}
Nov 03 20:05:53.732692 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:53.732612    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:54.439194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.434648    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:54.624642 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.624604    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" containerID="cri-o://c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d" gracePeriod=45
Nov 03 20:05:54.625569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.625549    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:54.680166 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.680134    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-openshift-ca\") pod \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") "
Nov 03 20:05:54.680419 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.680401    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-xks-ca\") pod \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") "
Nov 03 20:05:54.680516 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.680486    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-kube-api-access-qz6vf\" (UniqueName: \"kubernetes.io/projected/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-kube-api-access-qz6vf\") pod \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\" (UID: \"cfa1c1ee-7221-4575-b8c8-dd10e30aef62\") "
Nov 03 20:05:54.680596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.680564    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-openshift-ca" pod "cfa1c1ee-7221-4575-b8c8-dd10e30aef62" (UID: "cfa1c1ee-7221-4575-b8c8-dd10e30aef62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:54.681423 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.681133    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:54.683528 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.683506    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-xks-ca" pod "cfa1c1ee-7221-4575-b8c8-dd10e30aef62" (UID: "cfa1c1ee-7221-4575-b8c8-dd10e30aef62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:54.683659 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.683540    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-kube-api-access-qz6vf" pod "cfa1c1ee-7221-4575-b8c8-dd10e30aef62" (UID: "cfa1c1ee-7221-4575-b8c8-dd10e30aef62"). InnerVolumeSpecName "kube-api-access-qz6vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:54.783084 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.783046    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:54.783084 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.783083    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz6vf\" (UniqueName: \"kubernetes.io/projected/cfa1c1ee-7221-4575-b8c8-dd10e30aef62-kube-api-access-qz6vf\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:54.887629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.887602    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 20:05:54.985404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.985326    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tls-certs\") pod \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") "
Nov 03 20:05:54.985601 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.985407    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-home\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-home\") pod \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") "
Nov 03 20:05:54.985601 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.985479    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89c40ad-f434-42e9-a9b3-5c7311c380bd-kube-api-access-4v955\" (UniqueName: \"kubernetes.io/projected/d89c40ad-f434-42e9-a9b3-5c7311c380bd-kube-api-access-4v955\") pod \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") "
Nov 03 20:05:54.985736 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.985716    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-model-cache\") pod \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") "
Nov 03 20:05:54.985819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.985802    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-dshm\") pod \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") "
Nov 03 20:05:54.985911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.985895    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tmp-dir\") pod \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\" (UID: \"d89c40ad-f434-42e9-a9b3-5c7311c380bd\") "
Nov 03 20:05:54.985977 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.985913    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"]
Nov 03 20:05:54.986274 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.986248    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-model-cache" pod "d89c40ad-f434-42e9-a9b3-5c7311c380bd" (UID: "d89c40ad-f434-42e9-a9b3-5c7311c380bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:54.986441 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.986419    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tmp-dir" pod "d89c40ad-f434-42e9-a9b3-5c7311c380bd" (UID: "d89c40ad-f434-42e9-a9b3-5c7311c380bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:54.986981 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.986952    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cfa1c1ee-7221-4575-b8c8-dd10e30aef62" containerName="revoke-keys"
Nov 03 20:05:54.987136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.986983    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa1c1ee-7221-4575-b8c8-dd10e30aef62" containerName="revoke-keys"
Nov 03 20:05:54.987136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.987006    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main"
Nov 03 20:05:54.987136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.987015    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main"
Nov 03 20:05:54.987427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.987269    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa1c1ee-7221-4575-b8c8-dd10e30aef62" containerName="revoke-keys"
Nov 03 20:05:54.987427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.987285    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main"
Nov 03 20:05:54.987427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.987409    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:54.987625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.987433    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:54.987625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.987510    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-home" pod "d89c40ad-f434-42e9-a9b3-5c7311c380bd" (UID: "d89c40ad-f434-42e9-a9b3-5c7311c380bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:54.988437 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.988398    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tls-certs" pod "d89c40ad-f434-42e9-a9b3-5c7311c380bd" (UID: "d89c40ad-f434-42e9-a9b3-5c7311c380bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:54.989296 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.989273    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89c40ad-f434-42e9-a9b3-5c7311c380bd-kube-api-access-4v955" pod "d89c40ad-f434-42e9-a9b3-5c7311c380bd" (UID: "d89c40ad-f434-42e9-a9b3-5c7311c380bd"). InnerVolumeSpecName "kube-api-access-4v955". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:54.989390 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.989349    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-dshm" pod "d89c40ad-f434-42e9-a9b3-5c7311c380bd" (UID: "d89c40ad-f434-42e9-a9b3-5c7311c380bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:54.992080 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.992059    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:54.994847 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:54.994146    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"]
Nov 03 20:05:55.017474 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.017433    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"]
Nov 03 20:05:55.017644 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.017558    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:55.027249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.027159    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"]
Nov 03 20:05:55.088615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.088573    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd71da9a-qz2gv\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:55.088785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.088718    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a133ad1-58bjl\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:55.088828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.088803    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lrjk\" (UniqueName: \"kubernetes.io/projected/08310c0b-cba7-4218-bd85-3251607f6933-kube-api-access-2lrjk\") pod \"maas-api-revoke-sub-2a133ad1-58bjl\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:55.088987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.088972    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd71da9a-qz2gv\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:55.089027 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.088997    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljp49\" (UniqueName: \"kubernetes.io/projected/d317dc4c-09db-434c-be9b-923b1416e6af-kube-api-access-ljp49\") pod \"maas-api-revoke-sub-bd71da9a-qz2gv\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:55.089093 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.089081    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a133ad1-58bjl\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:55.089188 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.089164    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.089230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.089195    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89c40ad-f434-42e9-a9b3-5c7311c380bd-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.089230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.089209    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89c40ad-f434-42e9-a9b3-5c7311c380bd-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.089230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.089224    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v955\" (UniqueName: \"kubernetes.io/projected/d89c40ad-f434-42e9-a9b3-5c7311c380bd-kube-api-access-4v955\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.190924 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.190880    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd71da9a-qz2gv\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:55.191115 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.190937    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljp49\" (UniqueName: \"kubernetes.io/projected/d317dc4c-09db-434c-be9b-923b1416e6af-kube-api-access-ljp49\") pod \"maas-api-revoke-sub-bd71da9a-qz2gv\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:55.191115 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.191020    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a133ad1-58bjl\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:55.191115 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.191079    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd71da9a-qz2gv\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:55.191358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.191194    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a133ad1-58bjl\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:55.191358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.191248    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lrjk\" (UniqueName: \"kubernetes.io/projected/08310c0b-cba7-4218-bd85-3251607f6933-kube-api-access-2lrjk\") pod \"maas-api-revoke-sub-2a133ad1-58bjl\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:55.192360 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.192334    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a133ad1-58bjl\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:55.192859 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.192834    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd71da9a-qz2gv\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:55.193511 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.193484    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd71da9a-qz2gv\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:55.194350 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.194331    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a133ad1-58bjl\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:55.198962 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.198934    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljp49\" (UniqueName: \"kubernetes.io/projected/d317dc4c-09db-434c-be9b-923b1416e6af-kube-api-access-ljp49\") pod \"maas-api-revoke-sub-bd71da9a-qz2gv\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:55.199389 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.199369    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lrjk\" (UniqueName: \"kubernetes.io/projected/08310c0b-cba7-4218-bd85-3251607f6933-kube-api-access-2lrjk\") pod \"maas-api-revoke-sub-2a133ad1-58bjl\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:55.343201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.331676    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:55.371000 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.362337    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:55.387247 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.387077    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:55.422300 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.422266    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 20:05:55.476624 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.476599    4531 generic.go:356] "Generic (PLEG): container finished" podID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerID="c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d" exitCode=0
Nov 03 20:05:55.476734 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.476722    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"
Nov 03 20:05:55.487208 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.487150    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" event={"ID":"d89c40ad-f434-42e9-a9b3-5c7311c380bd","Type":"ContainerDied","Data":"c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d"}
Nov 03 20:05:55.487348 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.487235    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" event={"ID":"d89c40ad-f434-42e9-a9b3-5c7311c380bd","Type":"ContainerDied","Data":"70c8ba191ec242ad9625c8229f428e312a3d81f673473aabcdbc223c8ba6a587"}
Nov 03 20:05:55.487348 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.487260    4531 scope.go:122] "RemoveContainer" containerID="c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d"
Nov 03 20:05:55.492665 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.492639    4531 generic.go:356] "Generic (PLEG): container finished" podID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerID="4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651" exitCode=0
Nov 03 20:05:55.492783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.492751    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" event={"ID":"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce","Type":"ContainerDied","Data":"4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651"}
Nov 03 20:05:55.492851 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.492804    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8" event={"ID":"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce","Type":"ContainerDied","Data":"324aab5698f88b3132539aa1b04d8abb77dd19d1a1956e972f388d2045e5b965"}
Nov 03 20:05:55.492913 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.492897    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"
Nov 03 20:05:55.495106 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.495087    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"
Nov 03 20:05:55.495933 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.495153    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc" event={"ID":"cfa1c1ee-7221-4575-b8c8-dd10e30aef62","Type":"ContainerDied","Data":"31302fbc0f4b2e917ed94d494ca0ca0943f0b126e971d9afd863d359e660c0f5"}
Nov 03 20:05:55.496092 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.496078    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31302fbc0f4b2e917ed94d494ca0ca0943f0b126e971d9afd863d359e660c0f5"
Nov 03 20:05:55.499384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.499358    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-socket\") pod \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") "
Nov 03 20:05:55.499487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.499450    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-data\") pod \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") "
Nov 03 20:05:55.499554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.499507    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-kube-api-access-dfx57\" (UniqueName: \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-kube-api-access-dfx57\") pod \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") "
Nov 03 20:05:55.499628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.499567    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istiod-ca-cert\") pod \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") "
Nov 03 20:05:55.499692 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.499641    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-podinfo\") pod \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") "
Nov 03 20:05:55.499692 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.499675    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-token\" (UniqueName: \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-token\") pod \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") "
Nov 03 20:05:55.499804 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.499789    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-envoy\") pod \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") "
Nov 03 20:05:55.499864 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.499850    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-certs\") pod \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") "
Nov 03 20:05:55.499922 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.499897    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-credential-socket\") pod \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\" (UID: \"bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce\") "
Nov 03 20:05:55.501674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.500965    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-credential-socket" pod "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" (UID: "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:55.501674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.501385    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-socket" pod "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" (UID: "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:55.501674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.501530    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-data" pod "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" (UID: "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:55.501674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.501631    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-certs" pod "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" (UID: "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:55.505436 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.504449    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-envoy" pod "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" (UID: "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:55.510067 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.509527    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istiod-ca-cert" pod "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" (UID: "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:55.512911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.512889    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"]
Nov 03 20:05:55.515133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.514830    4531 scope.go:122] "RemoveContainer" containerID="c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d"
Nov 03 20:05:55.515280 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:55.515243    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d\": container with ID starting with c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d not found: ID does not exist" containerID="c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d"
Nov 03 20:05:55.515367 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.515274    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d"} err="failed to get container status \"c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d\": rpc error: code = NotFound desc = could not find container \"c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d\": container with ID starting with c09a3fa25c7e4aa5e21453784b929e14d3e664948a9e66b05c208fa9f642781d not found: ID does not exist"
Nov 03 20:05:55.515367 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.515289    4531 scope.go:122] "RemoveContainer" containerID="4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651"
Nov 03 20:05:55.518260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.518241    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm"]
Nov 03 20:05:55.519726 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.519539    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-podinfo" pod "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" (UID: "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:05:55.519726 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.519688    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-kube-api-access-dfx57" pod "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" (UID: "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce"). InnerVolumeSpecName "kube-api-access-dfx57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:55.523224 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.522335    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-token" pod "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" (UID: "bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:55.527594 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.527291    4531 scope.go:122] "RemoveContainer" containerID="cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66"
Nov 03 20:05:55.537833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.537817    4531 scope.go:122] "RemoveContainer" containerID="4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651"
Nov 03 20:05:55.538205 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:55.538164    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651\": container with ID starting with 4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651 not found: ID does not exist" containerID="4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651"
Nov 03 20:05:55.538280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.538201    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651"} err="failed to get container status \"4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651\": rpc error: code = NotFound desc = could not find container \"4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651\": container with ID starting with 4efd8272700efd4c138e54fbb4bbcb6b02ce2ce464950a9c3a5c387653ea4651 not found: ID does not exist"
Nov 03 20:05:55.538280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.538219    4531 scope.go:122] "RemoveContainer" containerID="cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66"
Nov 03 20:05:55.538497 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:55.538478    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66\": container with ID starting with cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66 not found: ID does not exist" containerID="cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66"
Nov 03 20:05:55.538541 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.538501    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66"} err="failed to get container status \"cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66\": rpc error: code = NotFound desc = could not find container \"cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66\": container with ID starting with cfa3fbf4d444a705bfa113bb4c1705998f2a6caf6780c2f8981fdf0bc6432b66 not found: ID does not exist"
Nov 03 20:05:55.602317 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.602284    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.602317 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.602318    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.602535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.602335    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.602535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.602351    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.602535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.602364    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.602535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.602376    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.602535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.602388    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfx57\" (UniqueName: \"kubernetes.io/projected/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-kube-api-access-dfx57\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.602535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.602402    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.602535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.602415    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.701381 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:05:55.701340    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd317dc4c_09db_434c_be9b_923b1416e6af.slice/crio-cce60db6c9bbfdaa73bcda82492ad432511cd9860cacbaa65c97438ab12cb975 WatchSource:0}: Error finding container cce60db6c9bbfdaa73bcda82492ad432511cd9860cacbaa65c97438ab12cb975: Status 404 returned error can't find the container with id cce60db6c9bbfdaa73bcda82492ad432511cd9860cacbaa65c97438ab12cb975
Nov 03 20:05:55.705753 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.704968    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"]
Nov 03 20:05:55.707242 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:05:55.707216    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08310c0b_cba7_4218_bd85_3251607f6933.slice/crio-5cb5f3f0b3f01f68f859f5d8f1f87f77f087d76e0940923e1eba2d5bbb28d851 WatchSource:0}: Error finding container 5cb5f3f0b3f01f68f859f5d8f1f87f77f087d76e0940923e1eba2d5bbb28d851: Status 404 returned error can't find the container with id 5cb5f3f0b3f01f68f859f5d8f1f87f77f087d76e0940923e1eba2d5bbb28d851
Nov 03 20:05:55.711408 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.711382    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"]
Nov 03 20:05:55.715737 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.715717    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:55.816337 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.816311    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"]
Nov 03 20:05:55.829904 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.822210    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-10572d-openshift-default-54488cd6dc-z62p8"]
Nov 03 20:05:55.840594 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.840558    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:05:55.840733 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.840620    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:05:55.856407 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:55.856373    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/premium-w1-a35658-kserve-6646c4894c-lj7dm" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Nov 03 20:05:56.215116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:56.215076    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:56.507384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:56.507343    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl" event={"ID":"08310c0b-cba7-4218-bd85-3251607f6933","Type":"ContainerStarted","Data":"aae25fd63be1f917299e25fd979c39e3573d674a10b2a9f72f064d8b8c37d8e8"}
Nov 03 20:05:56.507597 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:56.507410    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl" event={"ID":"08310c0b-cba7-4218-bd85-3251607f6933","Type":"ContainerStarted","Data":"5cb5f3f0b3f01f68f859f5d8f1f87f77f087d76e0940923e1eba2d5bbb28d851"}
Nov 03 20:05:56.508978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:56.508946    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv" event={"ID":"d317dc4c-09db-434c-be9b-923b1416e6af","Type":"ContainerStarted","Data":"4c3d70cd5af568c724d16d3df4cb7c127e2262fd35eb44e0d9ca9de9aaa6ee51"}
Nov 03 20:05:56.509106 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:56.508996    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv" event={"ID":"d317dc4c-09db-434c-be9b-923b1416e6af","Type":"ContainerStarted","Data":"cce60db6c9bbfdaa73bcda82492ad432511cd9860cacbaa65c97438ab12cb975"}
Nov 03 20:05:56.523196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:56.523136    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl" podStartSLOduration=2.523122533 podStartE2EDuration="2.523122533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:56.520397054 +0000 UTC m=+2069.790520351" watchObservedRunningTime="2026-11-03 20:05:56.523122533 +0000 UTC m=+2069.793245827"
Nov 03 20:05:56.535854 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:56.535809    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv" podStartSLOduration=2.5357962990000003 podStartE2EDuration="2.535796299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:56.533318976 +0000 UTC m=+2069.803442269" watchObservedRunningTime="2026-11-03 20:05:56.535796299 +0000 UTC m=+2069.805919592"
Nov 03 20:05:56.871849 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:56.871815    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" path="/var/lib/kubelet/pods/bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce/volumes"
Nov 03 20:05:56.872566 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:56.872548    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89c40ad-f434-42e9-a9b3-5c7311c380bd" path="/var/lib/kubelet/pods/d89c40ad-f434-42e9-a9b3-5c7311c380bd/volumes"
Nov 03 20:05:56.961564 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:56.960470    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:57.007157 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.007123    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:05:57.023492 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.023463    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-certs\") pod \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") "
Nov 03 20:05:57.023686 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.023557    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-podinfo\") pod \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") "
Nov 03 20:05:57.023686 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.023617    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istiod-ca-cert\") pod \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") "
Nov 03 20:05:57.023819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.023687    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-data\") pod \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") "
Nov 03 20:05:57.023880 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.023813    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-certs" pod "257503a5-c496-4bb6-a0fe-ed4295b9dacf" (UID: "257503a5-c496-4bb6-a0fe-ed4295b9dacf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:57.023880 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.023821    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-credential-socket\") pod \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") "
Nov 03 20:05:57.023992 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.023943    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-socket\") pod \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") "
Nov 03 20:05:57.024055 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.024033    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-data" pod "257503a5-c496-4bb6-a0fe-ed4295b9dacf" (UID: "257503a5-c496-4bb6-a0fe-ed4295b9dacf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:57.024206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.024051    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-credential-socket" pod "257503a5-c496-4bb6-a0fe-ed4295b9dacf" (UID: "257503a5-c496-4bb6-a0fe-ed4295b9dacf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:57.024206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.024063    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istiod-ca-cert" pod "257503a5-c496-4bb6-a0fe-ed4295b9dacf" (UID: "257503a5-c496-4bb6-a0fe-ed4295b9dacf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:57.025040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.024773    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-socket" pod "257503a5-c496-4bb6-a0fe-ed4295b9dacf" (UID: "257503a5-c496-4bb6-a0fe-ed4295b9dacf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:57.025040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.024814    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-envoy\") pod \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") "
Nov 03 20:05:57.025040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.024926    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-token\" (UniqueName: \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-token\") pod \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") "
Nov 03 20:05:57.025280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.025259    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-kube-api-access-lffdv\" (UniqueName: \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-kube-api-access-lffdv\") pod \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\" (UID: \"257503a5-c496-4bb6-a0fe-ed4295b9dacf\") "
Nov 03 20:05:57.026394 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.026365    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:57.026394 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.026393    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:57.026595 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.026407    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:57.026595 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.026420    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:57.026595 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.026436    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:57.027254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.027095    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-podinfo" pod "257503a5-c496-4bb6-a0fe-ed4295b9dacf" (UID: "257503a5-c496-4bb6-a0fe-ed4295b9dacf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:05:57.028040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.028012    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-token" pod "257503a5-c496-4bb6-a0fe-ed4295b9dacf" (UID: "257503a5-c496-4bb6-a0fe-ed4295b9dacf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:57.028272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.028248    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-envoy" pod "257503a5-c496-4bb6-a0fe-ed4295b9dacf" (UID: "257503a5-c496-4bb6-a0fe-ed4295b9dacf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:57.029373 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.029346    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-kube-api-access-lffdv" pod "257503a5-c496-4bb6-a0fe-ed4295b9dacf" (UID: "257503a5-c496-4bb6-a0fe-ed4295b9dacf"). InnerVolumeSpecName "kube-api-access-lffdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:57.128148 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.128113    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:57.128148 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.128147    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:57.128381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.128159    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lffdv\" (UniqueName: \"kubernetes.io/projected/257503a5-c496-4bb6-a0fe-ed4295b9dacf-kube-api-access-lffdv\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:57.128381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.128199    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257503a5-c496-4bb6-a0fe-ed4295b9dacf-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:57.519072 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.519030    4531 generic.go:356] "Generic (PLEG): container finished" podID="08310c0b-cba7-4218-bd85-3251607f6933" containerID="aae25fd63be1f917299e25fd979c39e3573d674a10b2a9f72f064d8b8c37d8e8" exitCode=0
Nov 03 20:05:57.519328 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.519115    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl" event={"ID":"08310c0b-cba7-4218-bd85-3251607f6933","Type":"ContainerDied","Data":"aae25fd63be1f917299e25fd979c39e3573d674a10b2a9f72f064d8b8c37d8e8"}
Nov 03 20:05:57.521069 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.521043    4531 generic.go:356] "Generic (PLEG): container finished" podID="d317dc4c-09db-434c-be9b-923b1416e6af" containerID="4c3d70cd5af568c724d16d3df4cb7c127e2262fd35eb44e0d9ca9de9aaa6ee51" exitCode=0
Nov 03 20:05:57.521238 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.521124    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv" event={"ID":"d317dc4c-09db-434c-be9b-923b1416e6af","Type":"ContainerDied","Data":"4c3d70cd5af568c724d16d3df4cb7c127e2262fd35eb44e0d9ca9de9aaa6ee51"}
Nov 03 20:05:57.522718 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.522697    4531 generic.go:356] "Generic (PLEG): container finished" podID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" containerID="7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb" exitCode=0
Nov 03 20:05:57.522832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.522725    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" event={"ID":"257503a5-c496-4bb6-a0fe-ed4295b9dacf","Type":"ContainerDied","Data":"7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb"}
Nov 03 20:05:57.522832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.522737    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"
Nov 03 20:05:57.522832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.522782    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9" event={"ID":"257503a5-c496-4bb6-a0fe-ed4295b9dacf","Type":"ContainerDied","Data":"79dce8458c6b32c7e2ed27b8028aeb7c50a036a55fe0557e1f5ee9fe062702dc"}
Nov 03 20:05:57.522832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.522827    4531 scope.go:122] "RemoveContainer" containerID="7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb"
Nov 03 20:05:57.536909 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.536888    4531 scope.go:122] "RemoveContainer" containerID="7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb"
Nov 03 20:05:57.537365 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:57.537331    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb\": container with ID starting with 7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb not found: ID does not exist" containerID="7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb"
Nov 03 20:05:57.537365 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.537373    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb"} err="failed to get container status \"7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb\": rpc error: code = NotFound desc = could not find container \"7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb\": container with ID starting with 7ced92925423bf301d91976ffd86a38afdb39e04d4d2740c4668f5e93b6ccdcb not found: ID does not exist"
Nov 03 20:05:57.553219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.551713    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"]
Nov 03 20:05:57.555991 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.555965    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-msqf9"]
Nov 03 20:05:57.664998 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.658738    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:57.848381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:57.848295    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:58.012315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.012246    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:58.285427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.285401    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 20:05:58.380456 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.380425    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istiod-ca-cert\") pod \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") "
Nov 03 20:05:58.380629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.380538    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-envoy\") pod \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") "
Nov 03 20:05:58.380629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.380570    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-podinfo\") pod \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") "
Nov 03 20:05:58.380629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.380601    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-socket\") pod \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") "
Nov 03 20:05:58.380798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.380691    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-token\" (UniqueName: \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-token\") pod \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") "
Nov 03 20:05:58.380798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.380769    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-data\") pod \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") "
Nov 03 20:05:58.380916 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.380841    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-kube-api-access-q2lwf\" (UniqueName: \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-kube-api-access-q2lwf\") pod \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") "
Nov 03 20:05:58.380976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.380920    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-certs\") pod \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") "
Nov 03 20:05:58.380976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.380960    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-credential-socket\") pod \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\" (UID: \"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8\") "
Nov 03 20:05:58.381248 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.380838    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istiod-ca-cert" pod "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" (UID: "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:58.381248 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.380998    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-socket" pod "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" (UID: "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:58.381375 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.381306    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-data" pod "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" (UID: "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:58.381375 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.381324    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-certs" pod "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" (UID: "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:58.381375 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.381340    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-credential-socket" pod "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" (UID: "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:58.381766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.381704    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.381883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.381773    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.381883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.381788    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.381883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.381804    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.381883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.381818    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.383373 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.383348    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-podinfo" pod "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" (UID: "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:05:58.383534 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.383512    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-kube-api-access-q2lwf" pod "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" (UID: "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8"). InnerVolumeSpecName "kube-api-access-q2lwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:58.383609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.383589    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-token" pod "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" (UID: "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:58.383937 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.383919    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-envoy" pod "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" (UID: "b17152ed-5f0b-4ef6-abb6-ff06e7a449e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:58.469849 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.452300    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:58.483886 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.483812    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.483886 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.483843    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.483886 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.483853    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.483886 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.483862    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2lwf\" (UniqueName: \"kubernetes.io/projected/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8-kube-api-access-q2lwf\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.533360 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.533325    4531 generic.go:356] "Generic (PLEG): container finished" podID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerID="82fefa89998d203254f5768fff02651ed935b615ecb2644d6ef2086f43341258" exitCode=0
Nov 03 20:05:58.533539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.533360    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" event={"ID":"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2","Type":"ContainerDied","Data":"82fefa89998d203254f5768fff02651ed935b615ecb2644d6ef2086f43341258"}
Nov 03 20:05:58.534832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.534807    4531 generic.go:356] "Generic (PLEG): container finished" podID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerID="6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83" exitCode=0
Nov 03 20:05:58.534950 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.534854    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"
Nov 03 20:05:58.534950 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.534887    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" event={"ID":"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8","Type":"ContainerDied","Data":"6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83"}
Nov 03 20:05:58.534950 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.534931    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps" event={"ID":"b17152ed-5f0b-4ef6-abb6-ff06e7a449e8","Type":"ContainerDied","Data":"78468aab5fb91158161925aeed2897d585da33b2df5e866f9208cc7c67e3f154"}
Nov 03 20:05:58.535096 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.534956    4531 scope.go:122] "RemoveContainer" containerID="6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83"
Nov 03 20:05:58.548021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.547990    4531 scope.go:122] "RemoveContainer" containerID="6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83"
Nov 03 20:05:58.548497 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:05:58.548454    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83\": container with ID starting with 6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83 not found: ID does not exist" containerID="6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83"
Nov 03 20:05:58.548601 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.548495    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83"} err="failed to get container status \"6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83\": rpc error: code = NotFound desc = could not find container \"6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83\": container with ID starting with 6f371d8cd4277b459ba117ed18fa07ea1034b63d9a81b3d41b6f96bb9b0bdb83 not found: ID does not exist"
Nov 03 20:05:58.563950 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.563884    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"]
Nov 03 20:05:58.567880 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.567856    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-ms9ps"]
Nov 03 20:05:58.772487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.772455    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:58.774686 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.774662    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 20:05:58.778480 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.778461    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:58.863347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.863294    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" path="/var/lib/kubelet/pods/257503a5-c496-4bb6-a0fe-ed4295b9dacf/volumes"
Nov 03 20:05:58.864164 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.864142    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" path="/var/lib/kubelet/pods/b17152ed-5f0b-4ef6-abb6-ff06e7a449e8/volumes"
Nov 03 20:05:58.892294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892224    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-kube-api-access-s9q2k\" (UniqueName: \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-kube-api-access-s9q2k\") pod \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") "
Nov 03 20:05:58.892441 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892300    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-xks-ca\") pod \"08310c0b-cba7-4218-bd85-3251607f6933\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") "
Nov 03 20:05:58.892441 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892362    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-openshift-ca\") pod \"08310c0b-cba7-4218-bd85-3251607f6933\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") "
Nov 03 20:05:58.892441 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892410    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-envoy\") pod \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") "
Nov 03 20:05:58.892566 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892499    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-socket\") pod \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") "
Nov 03 20:05:58.892566 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892529    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istiod-ca-cert\") pod \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") "
Nov 03 20:05:58.892660 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892563    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d317dc4c-09db-434c-be9b-923b1416e6af-kube-api-access-ljp49\" (UniqueName: \"kubernetes.io/projected/d317dc4c-09db-434c-be9b-923b1416e6af-kube-api-access-ljp49\") pod \"d317dc4c-09db-434c-be9b-923b1416e6af\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") "
Nov 03 20:05:58.892660 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892598    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-data\") pod \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") "
Nov 03 20:05:58.892772 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892667    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-podinfo\") pod \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") "
Nov 03 20:05:58.892772 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892712    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-xks-ca\") pod \"d317dc4c-09db-434c-be9b-923b1416e6af\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") "
Nov 03 20:05:58.892772 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892765    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-openshift-ca\") pod \"d317dc4c-09db-434c-be9b-923b1416e6af\" (UID: \"d317dc4c-09db-434c-be9b-923b1416e6af\") "
Nov 03 20:05:58.892936 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892769    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-openshift-ca" pod "08310c0b-cba7-4218-bd85-3251607f6933" (UID: "08310c0b-cba7-4218-bd85-3251607f6933"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:58.892936 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892833    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-certs\") pod \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") "
Nov 03 20:05:58.892936 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892770    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-socket" pod "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" (UID: "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:58.892936 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892860    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-credential-socket\") pod \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") "
Nov 03 20:05:58.893161 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892949    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-token\" (UniqueName: \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-token\") pod \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\" (UID: \"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2\") "
Nov 03 20:05:58.893161 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892967    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-data" pod "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" (UID: "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:58.893161 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.892977    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08310c0b-cba7-4218-bd85-3251607f6933-kube-api-access-2lrjk\" (UniqueName: \"kubernetes.io/projected/08310c0b-cba7-4218-bd85-3251607f6933-kube-api-access-2lrjk\") pod \"08310c0b-cba7-4218-bd85-3251607f6933\" (UID: \"08310c0b-cba7-4218-bd85-3251607f6933\") "
Nov 03 20:05:58.893359 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.893229    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-certs" pod "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" (UID: "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:58.893359 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.893275    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istiod-ca-cert" pod "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" (UID: "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:58.893638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.893611    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.893768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.893640    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.893768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.893656    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.893768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.893671    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.893768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.893685    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.893988 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.893763    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-credential-socket" pod "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" (UID: "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:58.894047 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.893983    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-openshift-ca" pod "d317dc4c-09db-434c-be9b-923b1416e6af" (UID: "d317dc4c-09db-434c-be9b-923b1416e6af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:58.895061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.895017    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-kube-api-access-s9q2k" pod "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" (UID: "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2"). InnerVolumeSpecName "kube-api-access-s9q2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:58.895435 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.895413    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-envoy" pod "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" (UID: "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:58.895672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.895641    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-xks-ca" pod "08310c0b-cba7-4218-bd85-3251607f6933" (UID: "08310c0b-cba7-4218-bd85-3251607f6933"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:58.895782 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.895678    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08310c0b-cba7-4218-bd85-3251607f6933-kube-api-access-2lrjk" pod "08310c0b-cba7-4218-bd85-3251607f6933" (UID: "08310c0b-cba7-4218-bd85-3251607f6933"). InnerVolumeSpecName "kube-api-access-2lrjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:58.895782 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.895688    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d317dc4c-09db-434c-be9b-923b1416e6af-kube-api-access-ljp49" pod "d317dc4c-09db-434c-be9b-923b1416e6af" (UID: "d317dc4c-09db-434c-be9b-923b1416e6af"). InnerVolumeSpecName "kube-api-access-ljp49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:58.896207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.896151    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-podinfo" pod "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" (UID: "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:05:58.896280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.896248    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-token" pod "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" (UID: "fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:58.896453 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.896436    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-xks-ca" pod "d317dc4c-09db-434c-be9b-923b1416e6af" (UID: "d317dc4c-09db-434c-be9b-923b1416e6af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:58.995145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.995101    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.995145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.995137    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljp49\" (UniqueName: \"kubernetes.io/projected/d317dc4c-09db-434c-be9b-923b1416e6af-kube-api-access-ljp49\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.995145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.995151    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.995450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.995163    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.995450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.995213    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d317dc4c-09db-434c-be9b-923b1416e6af-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.995450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.995226    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.995450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.995238    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.995450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.995251    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lrjk\" (UniqueName: \"kubernetes.io/projected/08310c0b-cba7-4218-bd85-3251607f6933-kube-api-access-2lrjk\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.995450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.995266    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9q2k\" (UniqueName: \"kubernetes.io/projected/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2-kube-api-access-s9q2k\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:58.995450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:58.995281    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08310c0b-cba7-4218-bd85-3251607f6933-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:59.202429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.202329    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:59.335084 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.335049    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:59.543710 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.543621    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"
Nov 03 20:05:59.543710 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.543636    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl" event={"ID":"fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2","Type":"ContainerDied","Data":"53d9a3bc0ed466cc69b481792177381d18a625d3875c93a7de3947dd9faf1d19"}
Nov 03 20:05:59.543948 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.543717    4531 scope.go:122] "RemoveContainer" containerID="82fefa89998d203254f5768fff02651ed935b615ecb2644d6ef2086f43341258"
Nov 03 20:05:59.547746 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.547713    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl" event={"ID":"08310c0b-cba7-4218-bd85-3251607f6933","Type":"ContainerDied","Data":"5cb5f3f0b3f01f68f859f5d8f1f87f77f087d76e0940923e1eba2d5bbb28d851"}
Nov 03 20:05:59.547993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.547779    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cb5f3f0b3f01f68f859f5d8f1f87f77f087d76e0940923e1eba2d5bbb28d851"
Nov 03 20:05:59.547993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.547726    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"
Nov 03 20:05:59.554259 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.549722    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv" event={"ID":"d317dc4c-09db-434c-be9b-923b1416e6af","Type":"ContainerDied","Data":"cce60db6c9bbfdaa73bcda82492ad432511cd9860cacbaa65c97438ab12cb975"}
Nov 03 20:05:59.554259 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.549766    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cce60db6c9bbfdaa73bcda82492ad432511cd9860cacbaa65c97438ab12cb975"
Nov 03 20:05:59.554259 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.549821    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"
Nov 03 20:05:59.578498 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.578470    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"]
Nov 03 20:05:59.583155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:05:59.583129    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-wmsvl"]
Nov 03 20:06:00.001606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:00.001574    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:00.717801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:00.717766    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:00.859575 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:00.859538    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" path="/var/lib/kubelet/pods/fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2/volumes"
Nov 03 20:06:00.887611 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:00.887577    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"]
Nov 03 20:06:01.018858 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:01.018757    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:01.280290 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:01.280158    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:01.453383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:01.453350    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:02.058028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:02.053166    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:02.294231 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:02.294200    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"]
Nov 03 20:06:02.404572 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:02.404536    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:02.543661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:02.543623    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:02.856416 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:02.834726    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:03.555206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:03.537225    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:04.295927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:04.295854    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:04.867382 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:04.867341    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:05.020211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:05.020121    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:05.675385 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:05.675348    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:05.888588 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:05.888541    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58" podUID="40903bcb-9b64-4233-8230-5b74407797fb" containerName="payload-processing" containerID="cri-o://07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313" gracePeriod=25
Nov 03 20:06:06.122250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.122147    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"]
Nov 03 20:06:06.122451 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.122427    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8" podUID="9377699d-5e72-4b01-966b-b9a4686c6859" containerName="maas-api" containerID="cri-o://7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083" gracePeriod=30
Nov 03 20:06:06.214291 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.214246    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:06.355973 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.355935    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:06.454197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.442405    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 20:06:06.466195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.465873    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-metrics-certs\") pod \"9377699d-5e72-4b01-966b-b9a4686c6859\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") "
Nov 03 20:06:06.466195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.465985    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9377699d-5e72-4b01-966b-b9a4686c6859-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9377699d-5e72-4b01-966b-b9a4686c6859-openshift-service-ca\") pod \"9377699d-5e72-4b01-966b-b9a4686c6859\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") "
Nov 03 20:06:06.466455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.466195    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-maas-api-tls\") pod \"9377699d-5e72-4b01-966b-b9a4686c6859\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") "
Nov 03 20:06:06.466455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.466325    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9377699d-5e72-4b01-966b-b9a4686c6859-kube-api-access-8lpkv\" (UniqueName: \"kubernetes.io/projected/9377699d-5e72-4b01-966b-b9a4686c6859-kube-api-access-8lpkv\") pod \"9377699d-5e72-4b01-966b-b9a4686c6859\" (UID: \"9377699d-5e72-4b01-966b-b9a4686c6859\") "
Nov 03 20:06:06.473037 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.467777    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9377699d-5e72-4b01-966b-b9a4686c6859-openshift-service-ca" pod "9377699d-5e72-4b01-966b-b9a4686c6859" (UID: "9377699d-5e72-4b01-966b-b9a4686c6859"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:06.493421 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.491523    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:06.549976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.549935    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-metrics-certs" pod "9377699d-5e72-4b01-966b-b9a4686c6859" (UID: "9377699d-5e72-4b01-966b-b9a4686c6859"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:06.550227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.550125    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9377699d-5e72-4b01-966b-b9a4686c6859-kube-api-access-8lpkv" pod "9377699d-5e72-4b01-966b-b9a4686c6859" (UID: "9377699d-5e72-4b01-966b-b9a4686c6859"). InnerVolumeSpecName "kube-api-access-8lpkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:06.563714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.563670    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-maas-api-tls" pod "9377699d-5e72-4b01-966b-b9a4686c6859" (UID: "9377699d-5e72-4b01-966b-b9a4686c6859"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:06.569424 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.569395    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.569424 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.569421    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpkv\" (UniqueName: \"kubernetes.io/projected/9377699d-5e72-4b01-966b-b9a4686c6859-kube-api-access-8lpkv\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.569424 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.569431    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9377699d-5e72-4b01-966b-b9a4686c6859-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.569725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.569440    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9377699d-5e72-4b01-966b-b9a4686c6859-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.616199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.615139    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58" podUID="40903bcb-9b64-4233-8230-5b74407797fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:06:06.627288 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.625949    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:06.641191 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.641148    4531 generic.go:356] "Generic (PLEG): container finished" podID="9377699d-5e72-4b01-966b-b9a4686c6859" containerID="7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083" exitCode=0
Nov 03 20:06:06.641419 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.641246    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8" event={"ID":"9377699d-5e72-4b01-966b-b9a4686c6859","Type":"ContainerDied","Data":"7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083"}
Nov 03 20:06:06.641419 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.641300    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"
Nov 03 20:06:06.641419 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.641322    4531 scope.go:122] "RemoveContainer" containerID="7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083"
Nov 03 20:06:06.641614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.641310    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8" event={"ID":"9377699d-5e72-4b01-966b-b9a4686c6859","Type":"ContainerDied","Data":"59c1b5d3a4b49cea76b54b7526325c8a3bf81145d539586cbcc4e11bbb22530e"}
Nov 03 20:06:06.653562 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.653542    4531 scope.go:122] "RemoveContainer" containerID="7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083"
Nov 03 20:06:06.653848 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:06.653808    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083\": container with ID starting with 7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083 not found: ID does not exist" containerID="7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083"
Nov 03 20:06:06.653926 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.653856    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083"} err="failed to get container status \"7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083\": rpc error: code = NotFound desc = could not find container \"7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083\": container with ID starting with 7a6f091d7f75099447e0b79c4d394380e2023f84fcb7ac034e156e0010157083 not found: ID does not exist"
Nov 03 20:06:06.666857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.666829    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"]
Nov 03 20:06:06.670391 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.670367    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6eb768-69d6d8568d-h2fd8"]
Nov 03 20:06:06.866436 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.866402    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9377699d-5e72-4b01-966b-b9a4686c6859" path="/var/lib/kubelet/pods/9377699d-5e72-4b01-966b-b9a4686c6859/volumes"
Nov 03 20:06:06.930811 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:06.930775    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.128897 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.128857    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"]
Nov 03 20:06:07.129560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129540    4531 cpu_manager.go:414] "containerMap: removing container" podUID="08310c0b-cba7-4218-bd85-3251607f6933" containerName="revoke-keys"
Nov 03 20:06:07.129560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129560    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08310c0b-cba7-4218-bd85-3251607f6933" containerName="revoke-keys"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129586    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129593    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129610    4531 cpu_manager.go:414] "containerMap: removing container" podUID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" containerName="istio-proxy"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129618    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" containerName="istio-proxy"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129632    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129639    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129649    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d317dc4c-09db-434c-be9b-923b1416e6af" containerName="revoke-keys"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129656    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d317dc4c-09db-434c-be9b-923b1416e6af" containerName="revoke-keys"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129669    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9377699d-5e72-4b01-966b-b9a4686c6859" containerName="maas-api"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129675    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9377699d-5e72-4b01-966b-b9a4686c6859" containerName="maas-api"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129711    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy"
Nov 03 20:06:07.129745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129718    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy"
Nov 03 20:06:07.130358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129895    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17152ed-5f0b-4ef6-abb6-ff06e7a449e8" containerName="istio-proxy"
Nov 03 20:06:07.130358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129906    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d317dc4c-09db-434c-be9b-923b1416e6af" containerName="revoke-keys"
Nov 03 20:06:07.130358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129919    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9377699d-5e72-4b01-966b-b9a4686c6859" containerName="maas-api"
Nov 03 20:06:07.130358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129933    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc42104f-5b1e-4ae2-9a4e-08fe160a4fa2" containerName="istio-proxy"
Nov 03 20:06:07.130358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129947    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy"
Nov 03 20:06:07.130358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129956    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="08310c0b-cba7-4218-bd85-3251607f6933" containerName="revoke-keys"
Nov 03 20:06:07.130358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129967    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy"
Nov 03 20:06:07.130358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.129976    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="257503a5-c496-4bb6-a0fe-ed4295b9dacf" containerName="istio-proxy"
Nov 03 20:06:07.134995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.134972    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:07.176198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.142881    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"]
Nov 03 20:06:07.176198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.143129    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:06:07.176198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.143220    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:06:07.176198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.159087    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"]
Nov 03 20:06:07.176198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.160042    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy"
Nov 03 20:06:07.176198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.160061    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc9c2e9-6cdc-435d-8ce8-4a1fae9e0bce" containerName="istio-proxy"
Nov 03 20:06:07.176198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.165415    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:07.176198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.167032    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"]
Nov 03 20:06:07.177552 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.177472    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c54243-tcznc\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:07.177679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.177579    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s6h8\" (UniqueName: \"kubernetes.io/projected/af9e3a33-2e59-4bbc-a431-fe482461a054-kube-api-access-6s6h8\") pod \"maas-api-revoke-sub-3b4b0eb7-2bl8v\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:07.177679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.177609    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b4b0eb7-2bl8v\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:07.177679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.177639    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swlhc\" (UniqueName: \"kubernetes.io/projected/8737e64b-f721-496c-8e5e-b61f14e55c80-kube-api-access-swlhc\") pod \"maas-api-revoke-sub-c6c54243-tcznc\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:07.177860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.177719    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c54243-tcznc\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:07.177911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.177871    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b4b0eb7-2bl8v\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:07.256205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.256121    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.279712 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.279680    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b4b0eb7-2bl8v\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:07.279874 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.279729    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c54243-tcznc\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:07.279874 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.279781    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s6h8\" (UniqueName: \"kubernetes.io/projected/af9e3a33-2e59-4bbc-a431-fe482461a054-kube-api-access-6s6h8\") pod \"maas-api-revoke-sub-3b4b0eb7-2bl8v\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:07.279874 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.279810    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b4b0eb7-2bl8v\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:07.279874 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.279849    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swlhc\" (UniqueName: \"kubernetes.io/projected/8737e64b-f721-496c-8e5e-b61f14e55c80-kube-api-access-swlhc\") pod \"maas-api-revoke-sub-c6c54243-tcznc\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:07.280090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.279949    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c54243-tcznc\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:07.280812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.280787    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b4b0eb7-2bl8v\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:07.280957 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.280924    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c54243-tcznc\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:07.282270 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.282246    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c54243-tcznc\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:07.282435 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.282418    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b4b0eb7-2bl8v\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:07.287360 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.287323    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s6h8\" (UniqueName: \"kubernetes.io/projected/af9e3a33-2e59-4bbc-a431-fe482461a054-kube-api-access-6s6h8\") pod \"maas-api-revoke-sub-3b4b0eb7-2bl8v\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:07.287449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.287337    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swlhc\" (UniqueName: \"kubernetes.io/projected/8737e64b-f721-496c-8e5e-b61f14e55c80-kube-api-access-swlhc\") pod \"maas-api-revoke-sub-c6c54243-tcznc\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:07.294777 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.294747    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq" podUID="987b5e63-c14d-473d-a5b6-bd5791338bce" containerName="payload-pre-processing" containerID="cri-o://519f38ac13e81ec067a9459c56a8df544ccb999024e558541a138ceea821f56c" gracePeriod=25
Nov 03 20:06:07.481207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.472734    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.481207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.473348    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:07.498211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.497391    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:07.597038 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.596542    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"]
Nov 03 20:06:07.605269 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.604765    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a35658-8ksnq"]
Nov 03 20:06:07.661649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.661549    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.673273 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.673238    4531 generic.go:356] "Generic (PLEG): container finished" podID="987b5e63-c14d-473d-a5b6-bd5791338bce" containerID="519f38ac13e81ec067a9459c56a8df544ccb999024e558541a138ceea821f56c" exitCode=0
Nov 03 20:06:07.673411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.673309    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq" event={"ID":"987b5e63-c14d-473d-a5b6-bd5791338bce","Type":"ContainerDied","Data":"519f38ac13e81ec067a9459c56a8df544ccb999024e558541a138ceea821f56c"}
Nov 03 20:06:07.812999 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.812893    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.822327 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.820992    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"]
Nov 03 20:06:07.824043 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:06:07.824004    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9e3a33_2e59_4bbc_a431_fe482461a054.slice/crio-ca28584e03425fdb26f35a44a05da1a4e506c1713030746323de983b5dabf0b2 WatchSource:0}: Error finding container ca28584e03425fdb26f35a44a05da1a4e506c1713030746323de983b5dabf0b2: Status 404 returned error can't find the container with id ca28584e03425fdb26f35a44a05da1a4e506c1713030746323de983b5dabf0b2
Nov 03 20:06:07.869919 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.869894    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 20:06:07.905124 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.905087    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987b5e63-c14d-473d-a5b6-bd5791338bce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987b5e63-c14d-473d-a5b6-bd5791338bce-plugins-config-volume\") pod \"987b5e63-c14d-473d-a5b6-bd5791338bce\" (UID: \"987b5e63-c14d-473d-a5b6-bd5791338bce\") "
Nov 03 20:06:07.905329 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.905290    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987b5e63-c14d-473d-a5b6-bd5791338bce-kube-api-access-5pmkd\" (UniqueName: \"kubernetes.io/projected/987b5e63-c14d-473d-a5b6-bd5791338bce-kube-api-access-5pmkd\") pod \"987b5e63-c14d-473d-a5b6-bd5791338bce\" (UID: \"987b5e63-c14d-473d-a5b6-bd5791338bce\") "
Nov 03 20:06:07.905475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.905451    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987b5e63-c14d-473d-a5b6-bd5791338bce-plugins-config-volume" pod "987b5e63-c14d-473d-a5b6-bd5791338bce" (UID: "987b5e63-c14d-473d-a5b6-bd5791338bce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:07.905944 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.905921    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987b5e63-c14d-473d-a5b6-bd5791338bce-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:07.907663 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:07.907639    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987b5e63-c14d-473d-a5b6-bd5791338bce-kube-api-access-5pmkd" pod "987b5e63-c14d-473d-a5b6-bd5791338bce" (UID: "987b5e63-c14d-473d-a5b6-bd5791338bce"). InnerVolumeSpecName "kube-api-access-5pmkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:08.007435 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.007400    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pmkd\" (UniqueName: \"kubernetes.io/projected/987b5e63-c14d-473d-a5b6-bd5791338bce-kube-api-access-5pmkd\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:08.044427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.044391    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"]
Nov 03 20:06:08.049554 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:06:08.049528    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8737e64b_f721_496c_8e5e_b61f14e55c80.slice/crio-b20aebd149f24901d8f8a4a4781603580762182aaa4f9f490cd2a1914357fba8 WatchSource:0}: Error finding container b20aebd149f24901d8f8a4a4781603580762182aaa4f9f490cd2a1914357fba8: Status 404 returned error can't find the container with id b20aebd149f24901d8f8a4a4781603580762182aaa4f9f490cd2a1914357fba8
Nov 03 20:06:08.178547 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.178511    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:08.684809 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.684764    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v" event={"ID":"af9e3a33-2e59-4bbc-a431-fe482461a054","Type":"ContainerStarted","Data":"b8030e363e398b89ed14fbc7c2ffe1884b90103b57be198597f94bfc37276c8a"}
Nov 03 20:06:08.684987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.684820    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v" event={"ID":"af9e3a33-2e59-4bbc-a431-fe482461a054","Type":"ContainerStarted","Data":"ca28584e03425fdb26f35a44a05da1a4e506c1713030746323de983b5dabf0b2"}
Nov 03 20:06:08.686616 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.686585    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc" event={"ID":"8737e64b-f721-496c-8e5e-b61f14e55c80","Type":"ContainerStarted","Data":"5e273430866b7964e8cf541eabf56a381bbf4545dd5afee50b8ca07dd4590f45"}
Nov 03 20:06:08.686767 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.686633    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc" event={"ID":"8737e64b-f721-496c-8e5e-b61f14e55c80","Type":"ContainerStarted","Data":"b20aebd149f24901d8f8a4a4781603580762182aaa4f9f490cd2a1914357fba8"}
Nov 03 20:06:08.688064 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.688046    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"
Nov 03 20:06:08.688155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.688072    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq" event={"ID":"987b5e63-c14d-473d-a5b6-bd5791338bce","Type":"ContainerDied","Data":"de3de870f6ed8ae42dd996bd6a4fe9c5c2937b0925ae94bc9ac69a4242594040"}
Nov 03 20:06:08.688155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.688120    4531 scope.go:122] "RemoveContainer" containerID="519f38ac13e81ec067a9459c56a8df544ccb999024e558541a138ceea821f56c"
Nov 03 20:06:08.704358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.704312    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v" podStartSLOduration=1.70430015 podStartE2EDuration="1.70430015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:08.702864374 +0000 UTC m=+2081.972987669" watchObservedRunningTime="2026-11-03 20:06:08.70430015 +0000 UTC m=+2081.974423444"
Nov 03 20:06:08.719819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.719777    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc" podStartSLOduration=1.719764968 podStartE2EDuration="1.719764968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:08.716133972 +0000 UTC m=+2081.986257265" watchObservedRunningTime="2026-11-03 20:06:08.719764968 +0000 UTC m=+2081.989888262"
Nov 03 20:06:08.730020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.729995    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"]
Nov 03 20:06:08.757203 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.754855    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6eb768-5f447fb8bc-bqjgq"]
Nov 03 20:06:08.775700 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.775660    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:08.859634 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.859601    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ceae64-9c23-4083-8d78-452dad1c3db0" path="/var/lib/kubelet/pods/29ceae64-9c23-4083-8d78-452dad1c3db0/volumes"
Nov 03 20:06:08.860379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:08.860357    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987b5e63-c14d-473d-a5b6-bd5791338bce" path="/var/lib/kubelet/pods/987b5e63-c14d-473d-a5b6-bd5791338bce/volumes"
Nov 03 20:06:09.535198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:09.534761    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:09.704051 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:09.704010    4531 generic.go:356] "Generic (PLEG): container finished" podID="8737e64b-f721-496c-8e5e-b61f14e55c80" containerID="5e273430866b7964e8cf541eabf56a381bbf4545dd5afee50b8ca07dd4590f45" exitCode=0
Nov 03 20:06:09.704267 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:09.704048    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc" event={"ID":"8737e64b-f721-496c-8e5e-b61f14e55c80","Type":"ContainerDied","Data":"5e273430866b7964e8cf541eabf56a381bbf4545dd5afee50b8ca07dd4590f45"}
Nov 03 20:06:09.709096 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:09.709072    4531 generic.go:356] "Generic (PLEG): container finished" podID="af9e3a33-2e59-4bbc-a431-fe482461a054" containerID="b8030e363e398b89ed14fbc7c2ffe1884b90103b57be198597f94bfc37276c8a" exitCode=0
Nov 03 20:06:09.709257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:09.709152    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v" event={"ID":"af9e3a33-2e59-4bbc-a431-fe482461a054","Type":"ContainerDied","Data":"b8030e363e398b89ed14fbc7c2ffe1884b90103b57be198597f94bfc37276c8a"}
Nov 03 20:06:10.288071 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.288000    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:10.887543 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.887515    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:10.924743 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.924709    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:10.960337 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.960301    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-openshift-ca\") pod \"8737e64b-f721-496c-8e5e-b61f14e55c80\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") "
Nov 03 20:06:10.960337 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.960342    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-openshift-ca\") pod \"af9e3a33-2e59-4bbc-a431-fe482461a054\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") "
Nov 03 20:06:10.960556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.960443    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-xks-ca\") pod \"8737e64b-f721-496c-8e5e-b61f14e55c80\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") "
Nov 03 20:06:10.960556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.960523    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9e3a33-2e59-4bbc-a431-fe482461a054-kube-api-access-6s6h8\" (UniqueName: \"kubernetes.io/projected/af9e3a33-2e59-4bbc-a431-fe482461a054-kube-api-access-6s6h8\") pod \"af9e3a33-2e59-4bbc-a431-fe482461a054\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") "
Nov 03 20:06:10.960665 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.960598    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-xks-ca\") pod \"af9e3a33-2e59-4bbc-a431-fe482461a054\" (UID: \"af9e3a33-2e59-4bbc-a431-fe482461a054\") "
Nov 03 20:06:10.960995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.960731    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8737e64b-f721-496c-8e5e-b61f14e55c80-kube-api-access-swlhc\" (UniqueName: \"kubernetes.io/projected/8737e64b-f721-496c-8e5e-b61f14e55c80-kube-api-access-swlhc\") pod \"8737e64b-f721-496c-8e5e-b61f14e55c80\" (UID: \"8737e64b-f721-496c-8e5e-b61f14e55c80\") "
Nov 03 20:06:10.960995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.960759    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-openshift-ca" pod "8737e64b-f721-496c-8e5e-b61f14e55c80" (UID: "8737e64b-f721-496c-8e5e-b61f14e55c80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:10.960995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.960782    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-openshift-ca" pod "af9e3a33-2e59-4bbc-a431-fe482461a054" (UID: "af9e3a33-2e59-4bbc-a431-fe482461a054"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:10.961848 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.961821    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:10.961964 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.961885    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:10.963052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.963022    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8737e64b-f721-496c-8e5e-b61f14e55c80-kube-api-access-swlhc" pod "8737e64b-f721-496c-8e5e-b61f14e55c80" (UID: "8737e64b-f721-496c-8e5e-b61f14e55c80"). InnerVolumeSpecName "kube-api-access-swlhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:10.963245 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.963221    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-xks-ca" pod "af9e3a33-2e59-4bbc-a431-fe482461a054" (UID: "af9e3a33-2e59-4bbc-a431-fe482461a054"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:10.963315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.963299    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-xks-ca" pod "8737e64b-f721-496c-8e5e-b61f14e55c80" (UID: "8737e64b-f721-496c-8e5e-b61f14e55c80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:10.963828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:10.963807    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9e3a33-2e59-4bbc-a431-fe482461a054-kube-api-access-6s6h8" pod "af9e3a33-2e59-4bbc-a431-fe482461a054" (UID: "af9e3a33-2e59-4bbc-a431-fe482461a054"). InnerVolumeSpecName "kube-api-access-6s6h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:11.063757 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.063718    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swlhc\" (UniqueName: \"kubernetes.io/projected/8737e64b-f721-496c-8e5e-b61f14e55c80-kube-api-access-swlhc\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.063757 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.063760    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8737e64b-f721-496c-8e5e-b61f14e55c80-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.063939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.063776    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s6h8\" (UniqueName: \"kubernetes.io/projected/af9e3a33-2e59-4bbc-a431-fe482461a054-kube-api-access-6s6h8\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.063939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.063792    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9e3a33-2e59-4bbc-a431-fe482461a054-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.129412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.129371    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:11.758446 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.756425    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:11.764281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.764088    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc" event={"ID":"8737e64b-f721-496c-8e5e-b61f14e55c80","Type":"ContainerDied","Data":"b20aebd149f24901d8f8a4a4781603580762182aaa4f9f490cd2a1914357fba8"}
Nov 03 20:06:11.764281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.764155    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b20aebd149f24901d8f8a4a4781603580762182aaa4f9f490cd2a1914357fba8"
Nov 03 20:06:11.764281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.764239    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"
Nov 03 20:06:11.771438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.771401    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"
Nov 03 20:06:11.771590 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.771406    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v" event={"ID":"af9e3a33-2e59-4bbc-a431-fe482461a054","Type":"ContainerDied","Data":"ca28584e03425fdb26f35a44a05da1a4e506c1713030746323de983b5dabf0b2"}
Nov 03 20:06:11.771590 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:11.771539    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca28584e03425fdb26f35a44a05da1a4e506c1713030746323de983b5dabf0b2"
Nov 03 20:06:12.408317 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:12.408258    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:12.925572 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:12.925536    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:13.148837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.148800    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:13.172725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.172688    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"]
Nov 03 20:06:13.173061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.173017    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy" containerID="cri-o://5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444" gracePeriod=30
Nov 03 20:06:13.304259 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.301188    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:13.795999 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.795966    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"]
Nov 03 20:06:13.796720 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.796697    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8737e64b-f721-496c-8e5e-b61f14e55c80" containerName="revoke-keys"
Nov 03 20:06:13.796720 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.796720    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8737e64b-f721-496c-8e5e-b61f14e55c80" containerName="revoke-keys"
Nov 03 20:06:13.796896 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.796776    4531 cpu_manager.go:414] "containerMap: removing container" podUID="af9e3a33-2e59-4bbc-a431-fe482461a054" containerName="revoke-keys"
Nov 03 20:06:13.796896 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.796781    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9e3a33-2e59-4bbc-a431-fe482461a054" containerName="revoke-keys"
Nov 03 20:06:13.796896 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.796793    4531 cpu_manager.go:414] "containerMap: removing container" podUID="987b5e63-c14d-473d-a5b6-bd5791338bce" containerName="payload-pre-processing"
Nov 03 20:06:13.796896 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.796798    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987b5e63-c14d-473d-a5b6-bd5791338bce" containerName="payload-pre-processing"
Nov 03 20:06:13.797043 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.796908    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9e3a33-2e59-4bbc-a431-fe482461a054" containerName="revoke-keys"
Nov 03 20:06:13.797043 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.796917    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="987b5e63-c14d-473d-a5b6-bd5791338bce" containerName="payload-pre-processing"
Nov 03 20:06:13.797043 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.796926    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8737e64b-f721-496c-8e5e-b61f14e55c80" containerName="revoke-keys"
Nov 03 20:06:13.801780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.801743    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:13.804836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.804810    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"]
Nov 03 20:06:13.804951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.804839    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:06:13.805352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.805331    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:06:13.934346 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.934306    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:13.934527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.934462    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsdbq\" (UniqueName: \"kubernetes.io/projected/65dbad91-76d4-4ad3-812d-419e09b156ea-kube-api-access-tsdbq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:13.934794 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:13.934753    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:14.011555 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.011353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:14.036947 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.036905    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:14.037390 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.037362    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:14.037483 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.037438    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsdbq\" (UniqueName: \"kubernetes.io/projected/65dbad91-76d4-4ad3-812d-419e09b156ea-kube-api-access-tsdbq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:14.038460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.038435    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:14.040268 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.040245    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:14.046439 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.046382    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsdbq\" (UniqueName: \"kubernetes.io/projected/65dbad91-76d4-4ad3-812d-419e09b156ea-kube-api-access-tsdbq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:14.120808 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.120768    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:14.356133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.356046    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:14.416557 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.416525    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"]
Nov 03 20:06:14.417635 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:06:14.417603    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65dbad91_76d4_4ad3_812d_419e09b156ea.slice/crio-3d5bd8f6b3d150dc9ef73bb9d92eda9d13df1c330566be5055b0dc4f2ac394b5 WatchSource:0}: Error finding container 3d5bd8f6b3d150dc9ef73bb9d92eda9d13df1c330566be5055b0dc4f2ac394b5: Status 404 returned error can't find the container with id 3d5bd8f6b3d150dc9ef73bb9d92eda9d13df1c330566be5055b0dc4f2ac394b5
Nov 03 20:06:14.489939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.489904    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:14.633191 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.633142    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"]
Nov 03 20:06:14.633452 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.633423    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" podUID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" containerName="istio-proxy" containerID="cri-o://eaa4cdc6f096264231d1fffc411e03e12cd7d929eb47f989a1300a6d6b2ccf87" gracePeriod=30
Nov 03 20:06:14.684854 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.684807    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:14.833691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:14.833609    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t" event={"ID":"65dbad91-76d4-4ad3-812d-419e09b156ea","Type":"ContainerStarted","Data":"3d5bd8f6b3d150dc9ef73bb9d92eda9d13df1c330566be5055b0dc4f2ac394b5"}
Nov 03 20:06:15.057245 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:15.051479    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:15.634592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:15.634466    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:15.842912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:15.842825    4531 generic.go:356] "Generic (PLEG): container finished" podID="65dbad91-76d4-4ad3-812d-419e09b156ea" containerID="954da0ed07416f83e03d229b4b87e32bda236804c75a9b189155c894751c8097" exitCode=0
Nov 03 20:06:15.843358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:15.842909    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t" event={"ID":"65dbad91-76d4-4ad3-812d-419e09b156ea","Type":"ContainerDied","Data":"954da0ed07416f83e03d229b4b87e32bda236804c75a9b189155c894751c8097"}
Nov 03 20:06:16.402001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:16.401966    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:16.553885 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:16.553856    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:16.624192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:16.623261    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58" podUID="40903bcb-9b64-4233-8230-5b74407797fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:06:16.677799 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:16.677710    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:17.049311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.049284    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:17.086081 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.086039    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:17.099622 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.099586    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-xks-ca\") pod \"65dbad91-76d4-4ad3-812d-419e09b156ea\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") "
Nov 03 20:06:17.099962 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.099935    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-openshift-ca\") pod \"65dbad91-76d4-4ad3-812d-419e09b156ea\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") "
Nov 03 20:06:17.100164 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.099978    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dbad91-76d4-4ad3-812d-419e09b156ea-kube-api-access-tsdbq\" (UniqueName: \"kubernetes.io/projected/65dbad91-76d4-4ad3-812d-419e09b156ea-kube-api-access-tsdbq\") pod \"65dbad91-76d4-4ad3-812d-419e09b156ea\" (UID: \"65dbad91-76d4-4ad3-812d-419e09b156ea\") "
Nov 03 20:06:17.100351 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.100328    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-openshift-ca" pod "65dbad91-76d4-4ad3-812d-419e09b156ea" (UID: "65dbad91-76d4-4ad3-812d-419e09b156ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:17.100866 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.100845    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:17.101894 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.101867    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-xks-ca" pod "65dbad91-76d4-4ad3-812d-419e09b156ea" (UID: "65dbad91-76d4-4ad3-812d-419e09b156ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:17.102024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.102007    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dbad91-76d4-4ad3-812d-419e09b156ea-kube-api-access-tsdbq" pod "65dbad91-76d4-4ad3-812d-419e09b156ea" (UID: "65dbad91-76d4-4ad3-812d-419e09b156ea"). InnerVolumeSpecName "kube-api-access-tsdbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:17.201862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.201824    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsdbq\" (UniqueName: \"kubernetes.io/projected/65dbad91-76d4-4ad3-812d-419e09b156ea-kube-api-access-tsdbq\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:17.201862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.201861    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dbad91-76d4-4ad3-812d-419e09b156ea-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:17.889162 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.889132    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"
Nov 03 20:06:17.889356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.889137    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t" event={"ID":"65dbad91-76d4-4ad3-812d-419e09b156ea","Type":"ContainerDied","Data":"3d5bd8f6b3d150dc9ef73bb9d92eda9d13df1c330566be5055b0dc4f2ac394b5"}
Nov 03 20:06:17.889356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:17.889224    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d5bd8f6b3d150dc9ef73bb9d92eda9d13df1c330566be5055b0dc4f2ac394b5"
Nov 03 20:06:18.466040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.465977    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 20:06:18.520789 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.520755    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-kube-api-access-6qcm8\" (UniqueName: \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-kube-api-access-6qcm8\") pod \"b63cceab-6dee-4689-8767-9a472d99b541\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") "
Nov 03 20:06:18.520789 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.520797    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-istio-token\" (UniqueName: \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-istio-token\") pod \"b63cceab-6dee-4689-8767-9a472d99b541\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") "
Nov 03 20:06:18.521004 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.520898    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-envoy\") pod \"b63cceab-6dee-4689-8767-9a472d99b541\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") "
Nov 03 20:06:18.521004 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.520993    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-socket\") pod \"b63cceab-6dee-4689-8767-9a472d99b541\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") "
Nov 03 20:06:18.521136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.521040    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-certs\") pod \"b63cceab-6dee-4689-8767-9a472d99b541\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") "
Nov 03 20:06:18.521136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.521077    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-credential-socket\") pod \"b63cceab-6dee-4689-8767-9a472d99b541\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") "
Nov 03 20:06:18.521270 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.521243    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-data\") pod \"b63cceab-6dee-4689-8767-9a472d99b541\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") "
Nov 03 20:06:18.521519 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.521310    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-socket" pod "b63cceab-6dee-4689-8767-9a472d99b541" (UID: "b63cceab-6dee-4689-8767-9a472d99b541"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:18.521632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.521318    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b63cceab-6dee-4689-8767-9a472d99b541-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63cceab-6dee-4689-8767-9a472d99b541-istiod-ca-cert\") pod \"b63cceab-6dee-4689-8767-9a472d99b541\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") "
Nov 03 20:06:18.521632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.521595    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-data" pod "b63cceab-6dee-4689-8767-9a472d99b541" (UID: "b63cceab-6dee-4689-8767-9a472d99b541"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:18.521632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.521609    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63cceab-6dee-4689-8767-9a472d99b541-istiod-ca-cert" pod "b63cceab-6dee-4689-8767-9a472d99b541" (UID: "b63cceab-6dee-4689-8767-9a472d99b541"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:18.521632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.521599    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b63cceab-6dee-4689-8767-9a472d99b541-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63cceab-6dee-4689-8767-9a472d99b541-istio-podinfo\") pod \"b63cceab-6dee-4689-8767-9a472d99b541\" (UID: \"b63cceab-6dee-4689-8767-9a472d99b541\") "
Nov 03 20:06:18.521828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.521379    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-certs" pod "b63cceab-6dee-4689-8767-9a472d99b541" (UID: "b63cceab-6dee-4689-8767-9a472d99b541"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:18.521828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.521483    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-credential-socket" pod "b63cceab-6dee-4689-8767-9a472d99b541" (UID: "b63cceab-6dee-4689-8767-9a472d99b541"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:18.522398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.522369    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:18.522551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.522401    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63cceab-6dee-4689-8767-9a472d99b541-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:18.522551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.522421    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:18.522551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.522436    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:18.522551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.522451    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:18.523719 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.523694    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-envoy" pod "b63cceab-6dee-4689-8767-9a472d99b541" (UID: "b63cceab-6dee-4689-8767-9a472d99b541"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:18.523882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.523867    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-kube-api-access-6qcm8" pod "b63cceab-6dee-4689-8767-9a472d99b541" (UID: "b63cceab-6dee-4689-8767-9a472d99b541"). InnerVolumeSpecName "kube-api-access-6qcm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:18.524149 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.524131    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b63cceab-6dee-4689-8767-9a472d99b541-istio-podinfo" pod "b63cceab-6dee-4689-8767-9a472d99b541" (UID: "b63cceab-6dee-4689-8767-9a472d99b541"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:06:18.524343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.524274    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-istio-token" pod "b63cceab-6dee-4689-8767-9a472d99b541" (UID: "b63cceab-6dee-4689-8767-9a472d99b541"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:18.625774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.625425    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63cceab-6dee-4689-8767-9a472d99b541-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:18.625774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.625473    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63cceab-6dee-4689-8767-9a472d99b541-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:18.625774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.625489    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qcm8\" (UniqueName: \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-kube-api-access-6qcm8\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:18.625774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.625503    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63cceab-6dee-4689-8767-9a472d99b541-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:18.685410 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.685340    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:18.898245 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.898210    4531 generic.go:356] "Generic (PLEG): container finished" podID="b63cceab-6dee-4689-8767-9a472d99b541" containerID="5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444" exitCode=0
Nov 03 20:06:18.898393 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.898262    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"
Nov 03 20:06:18.898393 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.898290    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" event={"ID":"b63cceab-6dee-4689-8767-9a472d99b541","Type":"ContainerDied","Data":"5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444"}
Nov 03 20:06:18.898393 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.898349    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf" event={"ID":"b63cceab-6dee-4689-8767-9a472d99b541","Type":"ContainerDied","Data":"b3c5cc652b35324bc46c14285dfe8888567de74ac6e556eb0c90cda5b3a21306"}
Nov 03 20:06:18.898513 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.898394    4531 scope.go:122] "RemoveContainer" containerID="5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444"
Nov 03 20:06:18.909422 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.909397    4531 scope.go:122] "RemoveContainer" containerID="1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577"
Nov 03 20:06:18.914830 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.914803    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"]
Nov 03 20:06:18.921258 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.921237    4531 scope.go:122] "RemoveContainer" containerID="5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444"
Nov 03 20:06:18.921579 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:18.921560    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444\": container with ID starting with 5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444 not found: ID does not exist" containerID="5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444"
Nov 03 20:06:18.921632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.921589    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444"} err="failed to get container status \"5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444\": rpc error: code = NotFound desc = could not find container \"5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444\": container with ID starting with 5bd69fdfe9e183cd789f97999236ca421b085a45c989a280904a7a2c11753444 not found: ID does not exist"
Nov 03 20:06:18.921632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.921608    4531 scope.go:122] "RemoveContainer" containerID="1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577"
Nov 03 20:06:18.921859 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:18.921844    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577\": container with ID starting with 1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577 not found: ID does not exist" containerID="1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577"
Nov 03 20:06:18.921908 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.921862    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577"} err="failed to get container status \"1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577\": rpc error: code = NotFound desc = could not find container \"1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577\": container with ID starting with 1915509c172006035df7ab571de7fe5827c077221c085ba0f05c0693e6a71577 not found: ID does not exist"
Nov 03 20:06:18.922026 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:18.922010    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6eb768-openshift-default-74584fc495-kg7bf"]
Nov 03 20:06:19.310535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:19.310472    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:19.908500 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:19.908416    4531 generic.go:356] "Generic (PLEG): container finished" podID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" containerID="eaa4cdc6f096264231d1fffc411e03e12cd7d929eb47f989a1300a6d6b2ccf87" exitCode=0
Nov 03 20:06:19.908918 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:19.908518    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" event={"ID":"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9","Type":"ContainerDied","Data":"eaa4cdc6f096264231d1fffc411e03e12cd7d929eb47f989a1300a6d6b2ccf87"}
Nov 03 20:06:19.908918 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:19.908561    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj" event={"ID":"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9","Type":"ContainerDied","Data":"09e90efd9bca4b4ff3e4a4c045172b80ec276f697395f830e95920a3d275b66b"}
Nov 03 20:06:19.908918 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:19.908578    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09e90efd9bca4b4ff3e4a4c045172b80ec276f697395f830e95920a3d275b66b"
Nov 03 20:06:19.916157 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:19.916127    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 20:06:20.052489 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.052456    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-kube-api-access-cvq52\" (UniqueName: \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-kube-api-access-cvq52\") pod \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") "
Nov 03 20:06:20.052776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.052526    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-credential-socket\") pod \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") "
Nov 03 20:06:20.052776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.052632    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-podinfo\") pod \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") "
Nov 03 20:06:20.052776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.052689    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-certs\") pod \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") "
Nov 03 20:06:20.052776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.052731    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-envoy\") pod \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") "
Nov 03 20:06:20.053020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.052800    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-token\" (UniqueName: \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-token\") pod \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") "
Nov 03 20:06:20.053020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.052888    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-credential-socket" pod "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" (UID: "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:20.053020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.052951    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-data\") pod \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") "
Nov 03 20:06:20.053020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.053006    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-socket\") pod \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") "
Nov 03 20:06:20.053252 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.053044    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istiod-ca-cert\") pod \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\" (UID: \"0d6a58ee-f52f-49ef-b0a7-d89dd10804d9\") "
Nov 03 20:06:20.053309 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.053282    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-data" pod "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" (UID: "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:20.053378 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.053302    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-socket" pod "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" (UID: "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:20.053563 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.053526    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-certs" pod "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" (UID: "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:20.053648 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.053622    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istiod-ca-cert" pod "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" (UID: "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:20.053779 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.053752    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.053779 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.053779    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.053953 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.053794    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.053953 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.053809    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.053953 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.053823    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.055673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.055650    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-podinfo" pod "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" (UID: "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:06:20.055751 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.055673    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-token" pod "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" (UID: "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:20.055751 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.055731    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-envoy" pod "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" (UID: "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:20.055751 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.055742    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-kube-api-access-cvq52" pod "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" (UID: "0d6a58ee-f52f-49ef-b0a7-d89dd10804d9"). InnerVolumeSpecName "kube-api-access-cvq52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:20.115197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.105785    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:20.157019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.156979    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.157019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.157018    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.157272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.157033    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvq52\" (UniqueName: \"kubernetes.io/projected/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-kube-api-access-cvq52\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.157272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.157048    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.573227 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:20.573167    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:06:20.886628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.886592    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:20.895371 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.895334    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b63cceab-6dee-4689-8767-9a472d99b541" path="/var/lib/kubelet/pods/b63cceab-6dee-4689-8767-9a472d99b541/volumes"
Nov 03 20:06:20.917030 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.917003    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"
Nov 03 20:06:20.934590 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.934542    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"]
Nov 03 20:06:20.938048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:20.938023    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-hzdkj"]
Nov 03 20:06:21.583608 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:21.583233    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:22.355339 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:22.355295    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:22.859110 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:22.859074    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" path="/var/lib/kubelet/pods/0d6a58ee-f52f-49ef-b0a7-d89dd10804d9/volumes"
Nov 03 20:06:23.094990 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:23.094952    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:23.807193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:23.801647    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:23.944334 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:23.944294    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"]
Nov 03 20:06:23.959254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:23.959214    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"]
Nov 03 20:06:24.030858 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:24.030823    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:24.158833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:24.158803    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:24.645211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:24.631809    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.064794 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.059247    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"]
Nov 03 20:06:25.064794 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.059544    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm" podUID="c08e67b7-ff2e-4845-9287-acad83fb619a" containerName="maas-api" containerID="cri-o://68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5" gracePeriod=30
Nov 03 20:06:25.144450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.144404    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.317581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.317464    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.418848 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.418731    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"]
Nov 03 20:06:25.423713 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.423682    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-b9cbea-xnq4t"]
Nov 03 20:06:25.446704 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.446680    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 20:06:25.464056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.464012    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.482190 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.482148    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08e67b7-ff2e-4845-9287-acad83fb619a-kube-api-access-mx4t8\" (UniqueName: \"kubernetes.io/projected/c08e67b7-ff2e-4845-9287-acad83fb619a-kube-api-access-mx4t8\") pod \"c08e67b7-ff2e-4845-9287-acad83fb619a\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") "
Nov 03 20:06:25.482384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.482236    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-metrics-certs\") pod \"c08e67b7-ff2e-4845-9287-acad83fb619a\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") "
Nov 03 20:06:25.482384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.482346    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-maas-api-tls\") pod \"c08e67b7-ff2e-4845-9287-acad83fb619a\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") "
Nov 03 20:06:25.482516 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.482501    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08e67b7-ff2e-4845-9287-acad83fb619a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08e67b7-ff2e-4845-9287-acad83fb619a-openshift-service-ca\") pod \"c08e67b7-ff2e-4845-9287-acad83fb619a\" (UID: \"c08e67b7-ff2e-4845-9287-acad83fb619a\") "
Nov 03 20:06:25.482905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.482874    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08e67b7-ff2e-4845-9287-acad83fb619a-openshift-service-ca" pod "c08e67b7-ff2e-4845-9287-acad83fb619a" (UID: "c08e67b7-ff2e-4845-9287-acad83fb619a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:25.483196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.483154    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08e67b7-ff2e-4845-9287-acad83fb619a-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:25.484670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.484649    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-metrics-certs" pod "c08e67b7-ff2e-4845-9287-acad83fb619a" (UID: "c08e67b7-ff2e-4845-9287-acad83fb619a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:25.484841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.484822    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08e67b7-ff2e-4845-9287-acad83fb619a-kube-api-access-mx4t8" pod "c08e67b7-ff2e-4845-9287-acad83fb619a" (UID: "c08e67b7-ff2e-4845-9287-acad83fb619a"). InnerVolumeSpecName "kube-api-access-mx4t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:25.485057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.485040    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-maas-api-tls" pod "c08e67b7-ff2e-4845-9287-acad83fb619a" (UID: "c08e67b7-ff2e-4845-9287-acad83fb619a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:25.601714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.601619    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:25.601714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.601658    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c08e67b7-ff2e-4845-9287-acad83fb619a-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:25.601714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.601672    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx4t8\" (UniqueName: \"kubernetes.io/projected/c08e67b7-ff2e-4845-9287-acad83fb619a-kube-api-access-mx4t8\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:25.609360 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.605994    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.957161 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.957126    4531 generic.go:356] "Generic (PLEG): container finished" podID="c08e67b7-ff2e-4845-9287-acad83fb619a" containerID="68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5" exitCode=0
Nov 03 20:06:25.957343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.957191    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"
Nov 03 20:06:25.957343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.957215    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm" event={"ID":"c08e67b7-ff2e-4845-9287-acad83fb619a","Type":"ContainerDied","Data":"68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5"}
Nov 03 20:06:25.957343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.957269    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm" event={"ID":"c08e67b7-ff2e-4845-9287-acad83fb619a","Type":"ContainerDied","Data":"bb6d38a85f826c71e27ed54e2ab08130862381c2d59d41456c61041aaee8e4b9"}
Nov 03 20:06:25.957343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.957291    4531 scope.go:122] "RemoveContainer" containerID="68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5"
Nov 03 20:06:25.959398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.959372    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.979123 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.979096    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"]
Nov 03 20:06:25.982405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:25.982383    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b9cbea-5b5b484f55-wlhfm"]
Nov 03 20:06:26.003418 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.003395    4531 scope.go:122] "RemoveContainer" containerID="68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5"
Nov 03 20:06:26.003746 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:26.003728    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5\": container with ID starting with 68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5 not found: ID does not exist" containerID="68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5"
Nov 03 20:06:26.003807 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.003755    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5"} err="failed to get container status \"68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5\": rpc error: code = NotFound desc = could not find container \"68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5\": container with ID starting with 68fafafe64d22996572469f2a48441f3532f4d1aab248f34dc9e6728b798ebd5 not found: ID does not exist"
Nov 03 20:06:26.162595 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.162524    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:26.163514 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.163497    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 20:06:26.209276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.209235    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40903bcb-9b64-4233-8230-5b74407797fb-kube-api-access-2cd9h\" (UniqueName: \"kubernetes.io/projected/40903bcb-9b64-4233-8230-5b74407797fb-kube-api-access-2cd9h\") pod \"40903bcb-9b64-4233-8230-5b74407797fb\" (UID: \"40903bcb-9b64-4233-8230-5b74407797fb\") "
Nov 03 20:06:26.209491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.209354    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40903bcb-9b64-4233-8230-5b74407797fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40903bcb-9b64-4233-8230-5b74407797fb-plugins-config-volume\") pod \"40903bcb-9b64-4233-8230-5b74407797fb\" (UID: \"40903bcb-9b64-4233-8230-5b74407797fb\") "
Nov 03 20:06:26.209915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.209883    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40903bcb-9b64-4233-8230-5b74407797fb-plugins-config-volume" pod "40903bcb-9b64-4233-8230-5b74407797fb" (UID: "40903bcb-9b64-4233-8230-5b74407797fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:26.210088 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.210065    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40903bcb-9b64-4233-8230-5b74407797fb-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:26.213203 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.213083    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40903bcb-9b64-4233-8230-5b74407797fb-kube-api-access-2cd9h" pod "40903bcb-9b64-4233-8230-5b74407797fb" (UID: "40903bcb-9b64-4233-8230-5b74407797fb"). InnerVolumeSpecName "kube-api-access-2cd9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:26.331067 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.329612    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cd9h\" (UniqueName: \"kubernetes.io/projected/40903bcb-9b64-4233-8230-5b74407797fb-kube-api-access-2cd9h\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:26.379253 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.379165    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:26.436346 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.436260    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"]
Nov 03 20:06:26.440905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.440876    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-8d4127-897fc"]
Nov 03 20:06:26.511262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.511227    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:26.644199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.643201    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:26.864250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.864142    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65dbad91-76d4-4ad3-812d-419e09b156ea" path="/var/lib/kubelet/pods/65dbad91-76d4-4ad3-812d-419e09b156ea/volumes"
Nov 03 20:06:26.864753 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.864737    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08e67b7-ff2e-4845-9287-acad83fb619a" path="/var/lib/kubelet/pods/c08e67b7-ff2e-4845-9287-acad83fb619a/volumes"
Nov 03 20:06:26.865143 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.865130    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa1c1ee-7221-4575-b8c8-dd10e30aef62" path="/var/lib/kubelet/pods/cfa1c1ee-7221-4575-b8c8-dd10e30aef62/volumes"
Nov 03 20:06:26.966808 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.966772    4531 generic.go:356] "Generic (PLEG): container finished" podID="40903bcb-9b64-4233-8230-5b74407797fb" containerID="07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313" exitCode=0
Nov 03 20:06:26.967039 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.966830    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"
Nov 03 20:06:26.967039 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.966857    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58" event={"ID":"40903bcb-9b64-4233-8230-5b74407797fb","Type":"ContainerDied","Data":"07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313"}
Nov 03 20:06:26.967039 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.966914    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58" event={"ID":"40903bcb-9b64-4233-8230-5b74407797fb","Type":"ContainerDied","Data":"4f58756c1fcdb192250b1848f75a5ca613666c4754146eb5ea1486c2ee8473c2"}
Nov 03 20:06:26.967039 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.966943    4531 scope.go:122] "RemoveContainer" containerID="07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313"
Nov 03 20:06:26.969152 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.969117    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:26.978700 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.978681    4531 scope.go:122] "RemoveContainer" containerID="07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313"
Nov 03 20:06:26.978963 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:26.978944    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313\": container with ID starting with 07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313 not found: ID does not exist" containerID="07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313"
Nov 03 20:06:26.979010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.978971    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313"} err="failed to get container status \"07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313\": rpc error: code = NotFound desc = could not find container \"07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313\": container with ID starting with 07c917d1f035f14acc7a2782546a0be69ef71cbad41ce5c1415ab5dae8ccd313 not found: ID does not exist"
Nov 03 20:06:26.987660 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.987636    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"]
Nov 03 20:06:26.991487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:26.991463    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8d4127-57d7c6bdc5-d4j58"]
Nov 03 20:06:27.438199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:27.432618    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:28.229791 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:28.229755    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:28.828200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:28.822629    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:28.867575 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:28.867539    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40903bcb-9b64-4233-8230-5b74407797fb" path="/var/lib/kubelet/pods/40903bcb-9b64-4233-8230-5b74407797fb/volumes"
Nov 03 20:06:28.945273 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:28.945227    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2" podUID="4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c" containerName="payload-processing" containerID="cri-o://a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3" gracePeriod=25
Nov 03 20:06:28.960795 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:28.960754    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb" podUID="e9f44648-77e8-408c-9193-9beb63991053" containerName="payload-pre-processing" containerID="cri-o://6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647" gracePeriod=25
Nov 03 20:06:29.245942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.245916    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 20:06:29.265093 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.265060    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-plugins-config-volume\") pod \"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c\" (UID: \"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c\") "
Nov 03 20:06:29.265382 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.265358    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-kube-api-access-qcstd\" (UniqueName: \"kubernetes.io/projected/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-kube-api-access-qcstd\") pod \"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c\" (UID: \"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c\") "
Nov 03 20:06:29.265526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.265480    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-plugins-config-volume" pod "4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c" (UID: "4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:29.265831 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.265801    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:29.267705 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.267677    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-kube-api-access-qcstd" pod "4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c" (UID: "4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c"). InnerVolumeSpecName "kube-api-access-qcstd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:29.290932 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.290906    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 20:06:29.366910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.366827    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f44648-77e8-408c-9193-9beb63991053-kube-api-access-fh99f\" (UniqueName: \"kubernetes.io/projected/e9f44648-77e8-408c-9193-9beb63991053-kube-api-access-fh99f\") pod \"e9f44648-77e8-408c-9193-9beb63991053\" (UID: \"e9f44648-77e8-408c-9193-9beb63991053\") "
Nov 03 20:06:29.367091 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.366989    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f44648-77e8-408c-9193-9beb63991053-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f44648-77e8-408c-9193-9beb63991053-plugins-config-volume\") pod \"e9f44648-77e8-408c-9193-9beb63991053\" (UID: \"e9f44648-77e8-408c-9193-9beb63991053\") "
Nov 03 20:06:29.367402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.367379    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f44648-77e8-408c-9193-9beb63991053-plugins-config-volume" pod "e9f44648-77e8-408c-9193-9beb63991053" (UID: "e9f44648-77e8-408c-9193-9beb63991053"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:29.367509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.367440    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcstd\" (UniqueName: \"kubernetes.io/projected/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c-kube-api-access-qcstd\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:29.368935 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.368915    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f44648-77e8-408c-9193-9beb63991053-kube-api-access-fh99f" pod "e9f44648-77e8-408c-9193-9beb63991053" (UID: "e9f44648-77e8-408c-9193-9beb63991053"). InnerVolumeSpecName "kube-api-access-fh99f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:29.468755 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.468713    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f44648-77e8-408c-9193-9beb63991053-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:29.468755 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.468746    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh99f\" (UniqueName: \"kubernetes.io/projected/e9f44648-77e8-408c-9193-9beb63991053-kube-api-access-fh99f\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:29.611614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.611576    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:29.995627 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.995593    4531 generic.go:356] "Generic (PLEG): container finished" podID="e9f44648-77e8-408c-9193-9beb63991053" containerID="6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647" exitCode=0
Nov 03 20:06:29.996167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.995641    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"
Nov 03 20:06:29.996167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.995672    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb" event={"ID":"e9f44648-77e8-408c-9193-9beb63991053","Type":"ContainerDied","Data":"6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647"}
Nov 03 20:06:29.996167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.995728    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb" event={"ID":"e9f44648-77e8-408c-9193-9beb63991053","Type":"ContainerDied","Data":"4e5e336547ddb3b63dd474c6c63be2a76b6b138221eb7e8208f4d25917194cff"}
Nov 03 20:06:29.996167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.995766    4531 scope.go:122] "RemoveContainer" containerID="6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647"
Nov 03 20:06:29.997439 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.997417    4531 generic.go:356] "Generic (PLEG): container finished" podID="4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c" containerID="a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3" exitCode=0
Nov 03 20:06:29.997563 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.997472    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"
Nov 03 20:06:29.997563 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.997508    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2" event={"ID":"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c","Type":"ContainerDied","Data":"a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3"}
Nov 03 20:06:29.997687 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:29.997575    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2" event={"ID":"4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c","Type":"ContainerDied","Data":"fac4913e8f0b1acbee53d44b5779a9f584a4c4f548d91387f4ed37babf47612e"}
Nov 03 20:06:30.010067 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.010039    4531 scope.go:122] "RemoveContainer" containerID="6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647"
Nov 03 20:06:30.010449 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:30.010423    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647\": container with ID starting with 6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647 not found: ID does not exist" containerID="6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647"
Nov 03 20:06:30.010538 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.010458    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647"} err="failed to get container status \"6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647\": rpc error: code = NotFound desc = could not find container \"6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647\": container with ID starting with 6f3c2c6092a8407600123d66d66c13287076ad7ed546f645f7c035e60587a647 not found: ID does not exist"
Nov 03 20:06:30.010538 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.010476    4531 scope.go:122] "RemoveContainer" containerID="a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3"
Nov 03 20:06:30.020133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.020112    4531 scope.go:122] "RemoveContainer" containerID="a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3"
Nov 03 20:06:30.020411 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:30.020392    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3\": container with ID starting with a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3 not found: ID does not exist" containerID="a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3"
Nov 03 20:06:30.020487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.020418    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3"} err="failed to get container status \"a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3\": rpc error: code = NotFound desc = could not find container \"a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3\": container with ID starting with a096957b72a4c80fa960c6930717b5f5b336c2f96a768019288e1182dbcd68c3 not found: ID does not exist"
Nov 03 20:06:30.024880 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.024858    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"]
Nov 03 20:06:30.030988 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.030965    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b9cbea-784c95f5cf-6lxvb"]
Nov 03 20:06:30.042450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.042422    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"]
Nov 03 20:06:30.045828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.045805    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b9cbea-76f64494bc-xjng2"]
Nov 03 20:06:30.321270 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.313733    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:30.862427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.862394    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c" path="/var/lib/kubelet/pods/4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c/volumes"
Nov 03 20:06:30.862924 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:30.862908    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f44648-77e8-408c-9193-9beb63991053" path="/var/lib/kubelet/pods/e9f44648-77e8-408c-9193-9beb63991053/volumes"
Nov 03 20:06:31.260419 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:31.258541    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:32.083197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:32.082790    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:32.504127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:32.504093    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:32.517127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:32.517092    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:06:32.545245 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:32.545209    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:06:32.736163 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:32.736130    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:32.982711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:32.982682    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:33.123787 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:33.123750    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:33.906396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:33.906360    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:34.314463 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:34.314364    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:34.451082 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:34.451049    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:34.584812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:34.584728    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:34.667636 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:34.667601    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"]
Nov 03 20:06:34.667886 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:34.667857    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy" containerID="cri-o://8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04" gracePeriod=30
Nov 03 20:06:34.673573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:34.673547    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"]
Nov 03 20:06:34.673748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:34.673729    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy" containerID="cri-o://b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999" gracePeriod=30
Nov 03 20:06:34.731922 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:34.731891    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:34.983959 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:34.983930    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:35.125673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.125638    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:35.220701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.220664    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"]
Nov 03 20:06:35.221483 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221461    4531 cpu_manager.go:414] "containerMap: removing container" podUID="65dbad91-76d4-4ad3-812d-419e09b156ea" containerName="revoke-keys"
Nov 03 20:06:35.221565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221487    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dbad91-76d4-4ad3-812d-419e09b156ea" containerName="revoke-keys"
Nov 03 20:06:35.221565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221504    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy"
Nov 03 20:06:35.221565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221512    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy"
Nov 03 20:06:35.221565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221525    4531 cpu_manager.go:414] "containerMap: removing container" podUID="40903bcb-9b64-4233-8230-5b74407797fb" containerName="payload-processing"
Nov 03 20:06:35.221565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221533    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40903bcb-9b64-4233-8230-5b74407797fb" containerName="payload-processing"
Nov 03 20:06:35.221565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221556    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e9f44648-77e8-408c-9193-9beb63991053" containerName="payload-pre-processing"
Nov 03 20:06:35.221565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221563    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f44648-77e8-408c-9193-9beb63991053" containerName="payload-pre-processing"
Nov 03 20:06:35.221827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221580    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c08e67b7-ff2e-4845-9287-acad83fb619a" containerName="maas-api"
Nov 03 20:06:35.221827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221587    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08e67b7-ff2e-4845-9287-acad83fb619a" containerName="maas-api"
Nov 03 20:06:35.221827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221617    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" containerName="istio-proxy"
Nov 03 20:06:35.221827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221624    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" containerName="istio-proxy"
Nov 03 20:06:35.221827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221637    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c" containerName="payload-processing"
Nov 03 20:06:35.221827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221645    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c" containerName="payload-processing"
Nov 03 20:06:35.221827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221788    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6a58ee-f52f-49ef-b0a7-d89dd10804d9" containerName="istio-proxy"
Nov 03 20:06:35.221827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221804    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4189fbf7-ecf2-44fa-8dc5-903a48a7ef7c" containerName="payload-processing"
Nov 03 20:06:35.221827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221818    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy"
Nov 03 20:06:35.221827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221829    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dbad91-76d4-4ad3-812d-419e09b156ea" containerName="revoke-keys"
Nov 03 20:06:35.222157 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221837    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="40903bcb-9b64-4233-8230-5b74407797fb" containerName="payload-processing"
Nov 03 20:06:35.222157 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221847    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f44648-77e8-408c-9193-9beb63991053" containerName="payload-pre-processing"
Nov 03 20:06:35.222157 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.221858    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08e67b7-ff2e-4845-9287-acad83fb619a" containerName="maas-api"
Nov 03 20:06:35.229001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.227567    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:35.232602 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.232574    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"]
Nov 03 20:06:35.272035 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.272001    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:35.272228 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.272114    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwvj\" (UniqueName: \"kubernetes.io/projected/20c474d5-b646-4802-acf6-8d57156c1165-kube-api-access-ptwvj\") pod \"maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:35.272374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.272354    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:35.291579 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.291553    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:06:35.291721 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.291593    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:06:35.373584 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.373555    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:35.373767 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.373596    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:35.373767 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.373639    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwvj\" (UniqueName: \"kubernetes.io/projected/20c474d5-b646-4802-acf6-8d57156c1165-kube-api-access-ptwvj\") pod \"maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:35.374407 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.374384    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:35.376003 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.375978    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:35.387189 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.382202    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwvj\" (UniqueName: \"kubernetes.io/projected/20c474d5-b646-4802-acf6-8d57156c1165-kube-api-access-ptwvj\") pod \"maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:35.409249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.409220    4531 scope.go:122] "RemoveContainer" containerID="eaa4cdc6f096264231d1fffc411e03e12cd7d929eb47f989a1300a6d6b2ccf87"
Nov 03 20:06:35.420034 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.420011    4531 scope.go:122] "RemoveContainer" containerID="a5e566870b104275c241ab2d21a85e6c9ecb3d13c06abd841be76dc6222af1a9"
Nov 03 20:06:35.551441 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.551410    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:35.822246 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.822023    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:35.843821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:35.843593    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"]
Nov 03 20:06:35.848222 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:06:35.848154    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c474d5_b646_4802_acf6_8d57156c1165.slice/crio-c7e459d5e5a82954aa2dbd872e6e81a629b937242804f1bf0b07779173028f85 WatchSource:0}: Error finding container c7e459d5e5a82954aa2dbd872e6e81a629b937242804f1bf0b07779173028f85: Status 404 returned error can't find the container with id c7e459d5e5a82954aa2dbd872e6e81a629b937242804f1bf0b07779173028f85
Nov 03 20:06:36.103637 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:36.103546    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv" event={"ID":"20c474d5-b646-4802-acf6-8d57156c1165","Type":"ContainerStarted","Data":"c7e459d5e5a82954aa2dbd872e6e81a629b937242804f1bf0b07779173028f85"}
Nov 03 20:06:36.389569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:36.389538    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:06:36.389734 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:36.389601    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:06:36.638192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:36.636110    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:37.112585 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:37.112494    4531 generic.go:356] "Generic (PLEG): container finished" podID="20c474d5-b646-4802-acf6-8d57156c1165" containerID="7edc56e526224d83fb6fd53968fffb84fa2fc1505168f7cba3c5b226fb2b9c2b" exitCode=0
Nov 03 20:06:37.112585 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:37.112547    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv" event={"ID":"20c474d5-b646-4802-acf6-8d57156c1165","Type":"ContainerDied","Data":"7edc56e526224d83fb6fd53968fffb84fa2fc1505168f7cba3c5b226fb2b9c2b"}
Nov 03 20:06:37.413206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:37.413142    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:38.239510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.228050    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:38.312112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.312089    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:38.423361 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.423315    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c474d5-b646-4802-acf6-8d57156c1165-kube-api-access-ptwvj\" (UniqueName: \"kubernetes.io/projected/20c474d5-b646-4802-acf6-8d57156c1165-kube-api-access-ptwvj\") pod \"20c474d5-b646-4802-acf6-8d57156c1165\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") "
Nov 03 20:06:38.423567 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.423368    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-openshift-ca\") pod \"20c474d5-b646-4802-acf6-8d57156c1165\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") "
Nov 03 20:06:38.423567 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.423486    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-xks-ca\") pod \"20c474d5-b646-4802-acf6-8d57156c1165\" (UID: \"20c474d5-b646-4802-acf6-8d57156c1165\") "
Nov 03 20:06:38.423801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.423773    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-openshift-ca" pod "20c474d5-b646-4802-acf6-8d57156c1165" (UID: "20c474d5-b646-4802-acf6-8d57156c1165"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:38.424206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.424163    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:38.425779 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.425750    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-xks-ca" pod "20c474d5-b646-4802-acf6-8d57156c1165" (UID: "20c474d5-b646-4802-acf6-8d57156c1165"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:38.425893 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.425836    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c474d5-b646-4802-acf6-8d57156c1165-kube-api-access-ptwvj" pod "20c474d5-b646-4802-acf6-8d57156c1165" (UID: "20c474d5-b646-4802-acf6-8d57156c1165"). InnerVolumeSpecName "kube-api-access-ptwvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:38.525241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.525202    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c474d5-b646-4802-acf6-8d57156c1165-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:38.525241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.525232    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwvj\" (UniqueName: \"kubernetes.io/projected/20c474d5-b646-4802-acf6-8d57156c1165-kube-api-access-ptwvj\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:38.923078 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:38.922876    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:39.128903 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:39.128880    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"
Nov 03 20:06:39.129074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:39.128911    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv" event={"ID":"20c474d5-b646-4802-acf6-8d57156c1165","Type":"ContainerDied","Data":"c7e459d5e5a82954aa2dbd872e6e81a629b937242804f1bf0b07779173028f85"}
Nov 03 20:06:39.129074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:39.128965    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e459d5e5a82954aa2dbd872e6e81a629b937242804f1bf0b07779173028f85"
Nov 03 20:06:39.559287 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:39.558863    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:39.981770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:39.981740    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 20:06:40.018219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.018160    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 20:06:40.045413 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.045382    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"]
Nov 03 20:06:40.061555 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.061521    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"]
Nov 03 20:06:40.069818 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.069789    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-socket\") pod \"c6e49e9f-167a-441d-824b-d8499196e409\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") "
Nov 03 20:06:40.070039 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.069848    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-kube-api-access-t4v4b\" (UniqueName: \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-kube-api-access-t4v4b\") pod \"c6e49e9f-167a-441d-824b-d8499196e409\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") "
Nov 03 20:06:40.070039 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.070012    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6e49e9f-167a-441d-824b-d8499196e409-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e49e9f-167a-441d-824b-d8499196e409-istio-podinfo\") pod \"c6e49e9f-167a-441d-824b-d8499196e409\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") "
Nov 03 20:06:40.070212 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.070149    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-data\") pod \"c6e49e9f-167a-441d-824b-d8499196e409\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") "
Nov 03 20:06:40.070212 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.070199    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-envoy\") pod \"c6e49e9f-167a-441d-824b-d8499196e409\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") "
Nov 03 20:06:40.070322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.070249    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-istio-token\" (UniqueName: \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-istio-token\") pod \"c6e49e9f-167a-441d-824b-d8499196e409\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") "
Nov 03 20:06:40.070428 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.070390    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-credential-socket\") pod \"c6e49e9f-167a-441d-824b-d8499196e409\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") "
Nov 03 20:06:40.070515 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.070440    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-data" pod "c6e49e9f-167a-441d-824b-d8499196e409" (UID: "c6e49e9f-167a-441d-824b-d8499196e409"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.070515 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.070449    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e49e9f-167a-441d-824b-d8499196e409-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e49e9f-167a-441d-824b-d8499196e409-istiod-ca-cert\") pod \"c6e49e9f-167a-441d-824b-d8499196e409\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") "
Nov 03 20:06:40.070515 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.070507    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-certs\") pod \"c6e49e9f-167a-441d-824b-d8499196e409\" (UID: \"c6e49e9f-167a-441d-824b-d8499196e409\") "
Nov 03 20:06:40.070701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.070677    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-credential-socket" pod "c6e49e9f-167a-441d-824b-d8499196e409" (UID: "c6e49e9f-167a-441d-824b-d8499196e409"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.070962 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.070941    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-certs" pod "c6e49e9f-167a-441d-824b-d8499196e409" (UID: "c6e49e9f-167a-441d-824b-d8499196e409"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.071083 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.071001    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-socket" pod "c6e49e9f-167a-441d-824b-d8499196e409" (UID: "c6e49e9f-167a-441d-824b-d8499196e409"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.071248 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.071227    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.071338 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.071253    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.071338 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.071255    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e49e9f-167a-441d-824b-d8499196e409-istiod-ca-cert" pod "c6e49e9f-167a-441d-824b-d8499196e409" (UID: "c6e49e9f-167a-441d-824b-d8499196e409"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:40.071338 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.071270    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.071338 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.071285    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.076538 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.076506    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6e49e9f-167a-441d-824b-d8499196e409-istio-podinfo" pod "c6e49e9f-167a-441d-824b-d8499196e409" (UID: "c6e49e9f-167a-441d-824b-d8499196e409"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:06:40.076663 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.076577    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-istio-token" pod "c6e49e9f-167a-441d-824b-d8499196e409" (UID: "c6e49e9f-167a-441d-824b-d8499196e409"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:40.077942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.077912    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-envoy" pod "c6e49e9f-167a-441d-824b-d8499196e409" (UID: "c6e49e9f-167a-441d-824b-d8499196e409"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.078083 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.078032    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-kube-api-access-t4v4b" pod "c6e49e9f-167a-441d-824b-d8499196e409" (UID: "c6e49e9f-167a-441d-824b-d8499196e409"). InnerVolumeSpecName "kube-api-access-t4v4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:40.140485 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.140444    4531 generic.go:356] "Generic (PLEG): container finished" podID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerID="b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999" exitCode=0
Nov 03 20:06:40.140713 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.140699    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"
Nov 03 20:06:40.141101 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.141078    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" event={"ID":"e68ef5cf-00e6-423c-92f6-ff8200f459b1","Type":"ContainerDied","Data":"b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999"}
Nov 03 20:06:40.141220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.141161    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z" event={"ID":"e68ef5cf-00e6-423c-92f6-ff8200f459b1","Type":"ContainerDied","Data":"f179c44b64d0d04e7e1e556c09fd58f67db5e4f1e5835378fec9215e613fe54a"}
Nov 03 20:06:40.141271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.141227    4531 scope.go:122] "RemoveContainer" containerID="b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999"
Nov 03 20:06:40.142844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.142817    4531 generic.go:356] "Generic (PLEG): container finished" podID="c6e49e9f-167a-441d-824b-d8499196e409" containerID="8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04" exitCode=0
Nov 03 20:06:40.142923 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.142873    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"
Nov 03 20:06:40.143028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.142874    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" event={"ID":"c6e49e9f-167a-441d-824b-d8499196e409","Type":"ContainerDied","Data":"8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04"}
Nov 03 20:06:40.143028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.143007    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf" event={"ID":"c6e49e9f-167a-441d-824b-d8499196e409","Type":"ContainerDied","Data":"5208b4609f59926f0da55e14e747b33d2e4579c85b9eb08ed04968735e999e47"}
Nov 03 20:06:40.152426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.152407    4531 scope.go:122] "RemoveContainer" containerID="9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004"
Nov 03 20:06:40.162375 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.162358    4531 scope.go:122] "RemoveContainer" containerID="b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999"
Nov 03 20:06:40.162665 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:40.162638    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999\": container with ID starting with b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999 not found: ID does not exist" containerID="b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999"
Nov 03 20:06:40.162746 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.162675    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999"} err="failed to get container status \"b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999\": rpc error: code = NotFound desc = could not find container \"b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999\": container with ID starting with b5a56fface5e68cbcc6ffc9b62dac6add93ae4e6fbc676fa3016fda23a0a1999 not found: ID does not exist"
Nov 03 20:06:40.162746 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.162700    4531 scope.go:122] "RemoveContainer" containerID="9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004"
Nov 03 20:06:40.163116 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:40.163083    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004\": container with ID starting with 9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004 not found: ID does not exist" containerID="9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004"
Nov 03 20:06:40.163278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.163124    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004"} err="failed to get container status \"9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004\": rpc error: code = NotFound desc = could not find container \"9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004\": container with ID starting with 9b00e633308c594ef897a92c9b744576b1651a0fa983402328cc998ce5e1e004 not found: ID does not exist"
Nov 03 20:06:40.163278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.163141    4531 scope.go:122] "RemoveContainer" containerID="8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04"
Nov 03 20:06:40.167095 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.167067    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"]
Nov 03 20:06:40.174555 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.174114    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-credential-socket\") pod \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") "
Nov 03 20:06:40.174555 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.174245    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-certs\") pod \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") "
Nov 03 20:06:40.174555 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.174284    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-envoy\") pod \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") "
Nov 03 20:06:40.174555 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.174514    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istiod-ca-cert\") pod \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") "
Nov 03 20:06:40.174970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.174780    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-credential-socket" pod "e68ef5cf-00e6-423c-92f6-ff8200f459b1" (UID: "e68ef5cf-00e6-423c-92f6-ff8200f459b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.174970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.174857    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istiod-ca-cert" pod "e68ef5cf-00e6-423c-92f6-ff8200f459b1" (UID: "e68ef5cf-00e6-423c-92f6-ff8200f459b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:40.174970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.174899    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-podinfo\") pod \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") "
Nov 03 20:06:40.174970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.174960    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-data\") pod \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") "
Nov 03 20:06:40.175232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.175056    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-socket\") pod \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") "
Nov 03 20:06:40.175232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.175096    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-token\" (UniqueName: \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-token\") pod \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") "
Nov 03 20:06:40.175232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.175125    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-kube-api-access-2mz2x\" (UniqueName: \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-kube-api-access-2mz2x\") pod \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\" (UID: \"e68ef5cf-00e6-423c-92f6-ff8200f459b1\") "
Nov 03 20:06:40.176018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.175688    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4v4b\" (UniqueName: \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-kube-api-access-t4v4b\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.176018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.175716    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e49e9f-167a-441d-824b-d8499196e409-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.176018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.175731    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e49e9f-167a-441d-824b-d8499196e409-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.176018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.175745    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e49e9f-167a-441d-824b-d8499196e409-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.176018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.175759    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.176018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.175773    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.176018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.175787    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e49e9f-167a-441d-824b-d8499196e409-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.176506 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.176313    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-socket" pod "e68ef5cf-00e6-423c-92f6-ff8200f459b1" (UID: "e68ef5cf-00e6-423c-92f6-ff8200f459b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.176669 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.176641    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-data" pod "e68ef5cf-00e6-423c-92f6-ff8200f459b1" (UID: "e68ef5cf-00e6-423c-92f6-ff8200f459b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.178225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.178164    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-kube-api-access-2mz2x" pod "e68ef5cf-00e6-423c-92f6-ff8200f459b1" (UID: "e68ef5cf-00e6-423c-92f6-ff8200f459b1"). InnerVolumeSpecName "kube-api-access-2mz2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:40.178425 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.178404    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-podinfo" pod "e68ef5cf-00e6-423c-92f6-ff8200f459b1" (UID: "e68ef5cf-00e6-423c-92f6-ff8200f459b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:06:40.178592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.178566    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-certs" pod "e68ef5cf-00e6-423c-92f6-ff8200f459b1" (UID: "e68ef5cf-00e6-423c-92f6-ff8200f459b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.179705 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.179678    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-token" pod "e68ef5cf-00e6-423c-92f6-ff8200f459b1" (UID: "e68ef5cf-00e6-423c-92f6-ff8200f459b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:40.180232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.180158    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-envoy" pod "e68ef5cf-00e6-423c-92f6-ff8200f459b1" (UID: "e68ef5cf-00e6-423c-92f6-ff8200f459b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.180985 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.180962    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6vfvf"]
Nov 03 20:06:40.182853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.182833    4531 scope.go:122] "RemoveContainer" containerID="8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04"
Nov 03 20:06:40.183221 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:40.183198    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04\": container with ID starting with 8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04 not found: ID does not exist" containerID="8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04"
Nov 03 20:06:40.183298 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.183233    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04"} err="failed to get container status \"8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04\": rpc error: code = NotFound desc = could not find container \"8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04\": container with ID starting with 8920720c0c534d292d59704fc7db9a78c7b9db59ceaaf52e502462a7104e2d04 not found: ID does not exist"
Nov 03 20:06:40.186680 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.186646    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:40.277239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.277203    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.277239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.277235    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.277239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.277245    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.277548 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.277257    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.277548 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.277271    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mz2x\" (UniqueName: \"kubernetes.io/projected/e68ef5cf-00e6-423c-92f6-ff8200f459b1-kube-api-access-2mz2x\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.277548 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.277281    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.277548 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.277288    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68ef5cf-00e6-423c-92f6-ff8200f459b1-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.326156 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.326116    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:40.463849 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.463802    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"]
Nov 03 20:06:40.467321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.467284    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-mwz5z"]
Nov 03 20:06:40.830640 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.830533    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:40.858337 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.858305    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e49e9f-167a-441d-824b-d8499196e409" path="/var/lib/kubelet/pods/c6e49e9f-167a-441d-824b-d8499196e409/volumes"
Nov 03 20:06:40.858846 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:40.858831    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" path="/var/lib/kubelet/pods/e68ef5cf-00e6-423c-92f6-ff8200f459b1/volumes"
Nov 03 20:06:41.053026 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:41.050204    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:41.813573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:41.813530    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:42.433756 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:42.433694    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:43.130824 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:43.130783    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:43.743200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:43.742283    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:44.023863 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:44.023772    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"]
Nov 03 20:06:44.027751 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:44.027723    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07655733-vdx7s"]
Nov 03 20:06:44.524254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:44.524209    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:44.858930 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:44.858847    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa40e8f-14ac-4070-8d1d-1276da8d481a" path="/var/lib/kubelet/pods/baa40e8f-14ac-4070-8d1d-1276da8d481a/volumes"
Nov 03 20:06:45.228056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:45.228019    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:45.386517 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:45.386482    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"]
Nov 03 20:06:45.402357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:45.401719    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"]
Nov 03 20:06:45.509726 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:45.509630    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:45.671391 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:45.671353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:45.942295 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:45.942256    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:46.621900 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:46.621865    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:46.832982 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:46.832950    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:47.241801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:47.241756    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:47.937191 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:47.937131    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:48.617287 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:48.617233    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:49.052078 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052039    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"]
Nov 03 20:06:49.052607 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052593    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy"
Nov 03 20:06:49.052655 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052608    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy"
Nov 03 20:06:49.052655 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052625    4531 cpu_manager.go:414] "containerMap: removing container" podUID="20c474d5-b646-4802-acf6-8d57156c1165" containerName="revoke-keys"
Nov 03 20:06:49.052655 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052630    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c474d5-b646-4802-acf6-8d57156c1165" containerName="revoke-keys"
Nov 03 20:06:49.052655 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052648    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy"
Nov 03 20:06:49.052655 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052654    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy"
Nov 03 20:06:49.052860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052662    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy"
Nov 03 20:06:49.052860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052666    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy"
Nov 03 20:06:49.052860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052764    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy"
Nov 03 20:06:49.052860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052774    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy"
Nov 03 20:06:49.052860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052782    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63cceab-6dee-4689-8767-9a472d99b541" containerName="istio-proxy"
Nov 03 20:06:49.052860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052790    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c474d5-b646-4802-acf6-8d57156c1165" containerName="revoke-keys"
Nov 03 20:06:49.052860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.052797    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e49e9f-167a-441d-824b-d8499196e409" containerName="istio-proxy"
Nov 03 20:06:49.058049 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.058025    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:49.061010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.060984    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"]
Nov 03 20:06:49.061950 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.061911    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy"
Nov 03 20:06:49.061950 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.061936    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68ef5cf-00e6-423c-92f6-ff8200f459b1" containerName="istio-proxy"
Nov 03 20:06:49.069023 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.068999    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"]
Nov 03 20:06:49.069143 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.069122    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:49.076879 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.076854    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:06:49.077014 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.076986    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:06:49.090526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.090498    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"]
Nov 03 20:06:49.110482 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.110447    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvczs\" (UniqueName: \"kubernetes.io/projected/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-kube-api-access-cvczs\") pod \"maas-api-revoke-sub-f913748c-4wx7q\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:49.110653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.110501    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6s6\" (UniqueName: \"kubernetes.io/projected/7e77260e-a4d1-4097-9859-8ece56379fd0-kube-api-access-gt6s6\") pod \"maas-api-revoke-sub-83e3d217-72xjc\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:49.110653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.110606    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83e3d217-72xjc\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:49.110787 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.110709    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f913748c-4wx7q\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:49.110787 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.110780    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f913748c-4wx7q\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:49.110902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.110875    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83e3d217-72xjc\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:49.211632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.211598    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f913748c-4wx7q\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:49.211819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.211708    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f913748c-4wx7q\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:49.211819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.211756    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83e3d217-72xjc\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:49.211954 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.211851    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvczs\" (UniqueName: \"kubernetes.io/projected/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-kube-api-access-cvczs\") pod \"maas-api-revoke-sub-f913748c-4wx7q\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:49.211954 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.211939    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6s6\" (UniqueName: \"kubernetes.io/projected/7e77260e-a4d1-4097-9859-8ece56379fd0-kube-api-access-gt6s6\") pod \"maas-api-revoke-sub-83e3d217-72xjc\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:49.212231 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.212202    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83e3d217-72xjc\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:49.212536 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.212491    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f913748c-4wx7q\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:49.212922 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.212898    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83e3d217-72xjc\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:49.214394 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.214369    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83e3d217-72xjc\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:49.214524 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.214507    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f913748c-4wx7q\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:49.219465 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.219441    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvczs\" (UniqueName: \"kubernetes.io/projected/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-kube-api-access-cvczs\") pod \"maas-api-revoke-sub-f913748c-4wx7q\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:49.219712 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.219685    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6s6\" (UniqueName: \"kubernetes.io/projected/7e77260e-a4d1-4097-9859-8ece56379fd0-kube-api-access-gt6s6\") pod \"maas-api-revoke-sub-83e3d217-72xjc\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:49.367756 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.367664    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:49.375249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.375216    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:49.386320 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.386291    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:49.665837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.665807    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"]
Nov 03 20:06:49.666567 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:06:49.666543    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a77b64_e0cd_4766_a0f1_e0b25cf73e54.slice/crio-b9497036134e1a4346e5d05051a04d5efcfab36435c76f75c247dda1543cb3b6 WatchSource:0}: Error finding container b9497036134e1a4346e5d05051a04d5efcfab36435c76f75c247dda1543cb3b6: Status 404 returned error can't find the container with id b9497036134e1a4346e5d05051a04d5efcfab36435c76f75c247dda1543cb3b6
Nov 03 20:06:49.672553 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:49.672220    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"]
Nov 03 20:06:49.674144 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:06:49.674094    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e77260e_a4d1_4097_9859_8ece56379fd0.slice/crio-b6f0d5441cf50f43cd81d0deedca9a0696729a41d534de981514c51787b6aede WatchSource:0}: Error finding container b6f0d5441cf50f43cd81d0deedca9a0696729a41d534de981514c51787b6aede: Status 404 returned error can't find the container with id b6f0d5441cf50f43cd81d0deedca9a0696729a41d534de981514c51787b6aede
Nov 03 20:06:50.049218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:50.037599    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:50.235055 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:50.234953    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q" event={"ID":"75a77b64-e0cd-4766-a0f1-e0b25cf73e54","Type":"ContainerStarted","Data":"813b047ecb68b9b8f94fbe09e90e5985a2a2cfc35b117ef711abcaaf41512c00"}
Nov 03 20:06:50.235055 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:50.235022    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q" event={"ID":"75a77b64-e0cd-4766-a0f1-e0b25cf73e54","Type":"ContainerStarted","Data":"b9497036134e1a4346e5d05051a04d5efcfab36435c76f75c247dda1543cb3b6"}
Nov 03 20:06:50.236769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:50.236743    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc" event={"ID":"7e77260e-a4d1-4097-9859-8ece56379fd0","Type":"ContainerStarted","Data":"ec75c5deee30f05491dddad9caca6eddcee342a61d1ff9f3d7e78aa695b3769f"}
Nov 03 20:06:50.236915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:50.236787    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc" event={"ID":"7e77260e-a4d1-4097-9859-8ece56379fd0","Type":"ContainerStarted","Data":"b6f0d5441cf50f43cd81d0deedca9a0696729a41d534de981514c51787b6aede"}
Nov 03 20:06:50.250395 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:50.250336    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q" podStartSLOduration=1.25032227 podStartE2EDuration="1.25032227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:50.248843231 +0000 UTC m=+2123.518966525" watchObservedRunningTime="2026-11-03 20:06:50.25032227 +0000 UTC m=+2123.520445565"
Nov 03 20:06:50.263413 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:50.263352    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc" podStartSLOduration=1.26333719 podStartE2EDuration="1.26333719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:50.261523768 +0000 UTC m=+2123.531647065" watchObservedRunningTime="2026-11-03 20:06:50.26333719 +0000 UTC m=+2123.533460525"
Nov 03 20:06:50.387128 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:50.387086    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94" podUID="752cf009-4d23-4820-ae58-b3ad21317e57" containerName="payload-processing" containerID="cri-o://02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b" gracePeriod=25
Nov 03 20:06:50.402688 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:50.402641    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m" podUID="db869375-d3d6-4043-b87f-5df736ed2939" containerName="payload-pre-processing" containerID="cri-o://02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16" gracePeriod=25
Nov 03 20:06:50.574089 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:06:50.574044    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:06:50.819930 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:50.817678    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:51.244935 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:51.244902    4531 generic.go:356] "Generic (PLEG): container finished" podID="75a77b64-e0cd-4766-a0f1-e0b25cf73e54" containerID="813b047ecb68b9b8f94fbe09e90e5985a2a2cfc35b117ef711abcaaf41512c00" exitCode=0
Nov 03 20:06:51.245137 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:51.244987    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q" event={"ID":"75a77b64-e0cd-4766-a0f1-e0b25cf73e54","Type":"ContainerDied","Data":"813b047ecb68b9b8f94fbe09e90e5985a2a2cfc35b117ef711abcaaf41512c00"}
Nov 03 20:06:51.246466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:51.246439    4531 generic.go:356] "Generic (PLEG): container finished" podID="7e77260e-a4d1-4097-9859-8ece56379fd0" containerID="ec75c5deee30f05491dddad9caca6eddcee342a61d1ff9f3d7e78aa695b3769f" exitCode=0
Nov 03 20:06:51.246589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:51.246517    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc" event={"ID":"7e77260e-a4d1-4097-9859-8ece56379fd0","Type":"ContainerDied","Data":"ec75c5deee30f05491dddad9caca6eddcee342a61d1ff9f3d7e78aa695b3769f"}
Nov 03 20:06:51.416436 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:51.416403    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:52.058458 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.058250    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:52.427841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.427816    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:52.432290 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.432268    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:52.465507 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.465477    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-openshift-ca\") pod \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") "
Nov 03 20:06:52.465703 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.465543    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-xks-ca\") pod \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") "
Nov 03 20:06:52.465703 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.465628    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e77260e-a4d1-4097-9859-8ece56379fd0-kube-api-access-gt6s6\" (UniqueName: \"kubernetes.io/projected/7e77260e-a4d1-4097-9859-8ece56379fd0-kube-api-access-gt6s6\") pod \"7e77260e-a4d1-4097-9859-8ece56379fd0\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") "
Nov 03 20:06:52.465844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.465773    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-kube-api-access-cvczs\" (UniqueName: \"kubernetes.io/projected/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-kube-api-access-cvczs\") pod \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\" (UID: \"75a77b64-e0cd-4766-a0f1-e0b25cf73e54\") "
Nov 03 20:06:52.465929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.465902    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-openshift-ca" pod "75a77b64-e0cd-4766-a0f1-e0b25cf73e54" (UID: "75a77b64-e0cd-4766-a0f1-e0b25cf73e54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:52.466415 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.466388    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-openshift-ca\") pod \"7e77260e-a4d1-4097-9859-8ece56379fd0\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") "
Nov 03 20:06:52.467055 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.466929    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-openshift-ca" pod "7e77260e-a4d1-4097-9859-8ece56379fd0" (UID: "7e77260e-a4d1-4097-9859-8ece56379fd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:52.467351 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.467006    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-xks-ca\") pod \"7e77260e-a4d1-4097-9859-8ece56379fd0\" (UID: \"7e77260e-a4d1-4097-9859-8ece56379fd0\") "
Nov 03 20:06:52.468089 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.468064    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:52.468317 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.468298    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:52.468530 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.468507    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-xks-ca" pod "75a77b64-e0cd-4766-a0f1-e0b25cf73e54" (UID: "75a77b64-e0cd-4766-a0f1-e0b25cf73e54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:52.468825 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.468803    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-kube-api-access-cvczs" pod "75a77b64-e0cd-4766-a0f1-e0b25cf73e54" (UID: "75a77b64-e0cd-4766-a0f1-e0b25cf73e54"). InnerVolumeSpecName "kube-api-access-cvczs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:52.470599 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.470571    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-xks-ca" pod "7e77260e-a4d1-4097-9859-8ece56379fd0" (UID: "7e77260e-a4d1-4097-9859-8ece56379fd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:52.470752 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.470734    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e77260e-a4d1-4097-9859-8ece56379fd0-kube-api-access-gt6s6" pod "7e77260e-a4d1-4097-9859-8ece56379fd0" (UID: "7e77260e-a4d1-4097-9859-8ece56379fd0"). InnerVolumeSpecName "kube-api-access-gt6s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:52.569986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.569904    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e77260e-a4d1-4097-9859-8ece56379fd0-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:52.569986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.569940    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:52.569986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.569951    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6s6\" (UniqueName: \"kubernetes.io/projected/7e77260e-a4d1-4097-9859-8ece56379fd0-kube-api-access-gt6s6\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:52.569986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.569962    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvczs\" (UniqueName: \"kubernetes.io/projected/75a77b64-e0cd-4766-a0f1-e0b25cf73e54-kube-api-access-cvczs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:52.670134 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:52.665686    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:53.261837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:53.261807    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"
Nov 03 20:06:53.261837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:53.261818    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q" event={"ID":"75a77b64-e0cd-4766-a0f1-e0b25cf73e54","Type":"ContainerDied","Data":"b9497036134e1a4346e5d05051a04d5efcfab36435c76f75c247dda1543cb3b6"}
Nov 03 20:06:53.262364 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:53.261871    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9497036134e1a4346e5d05051a04d5efcfab36435c76f75c247dda1543cb3b6"
Nov 03 20:06:53.263424 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:53.263402    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"
Nov 03 20:06:53.263424 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:53.263418    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc" event={"ID":"7e77260e-a4d1-4097-9859-8ece56379fd0","Type":"ContainerDied","Data":"b6f0d5441cf50f43cd81d0deedca9a0696729a41d534de981514c51787b6aede"}
Nov 03 20:06:53.263566 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:53.263456    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6f0d5441cf50f43cd81d0deedca9a0696729a41d534de981514c51787b6aede"
Nov 03 20:06:53.323722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:53.323674    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:53.958093 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:53.958049    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:54.617048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:54.616297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:55.068269 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.068229    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main" containerID="cri-o://1592c95b3a604b9241ab73b1405d3ff745ef6025436a5e04f51d33b30fce89f2" gracePeriod=45
Nov 03 20:06:55.084379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.084348    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" containerID="cri-o://2dca25c01e61f165c79beca85476e3d37c1a967f15be6a27a396bf21327e6959" gracePeriod=45
Nov 03 20:06:55.136296 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.136251    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": dial tcp 10.133.0.49:8000: connect: connection refused"
Nov 03 20:06:55.304995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.297152    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:55.325468 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.325388    4531 generic.go:356] "Generic (PLEG): container finished" podID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerID="1592c95b3a604b9241ab73b1405d3ff745ef6025436a5e04f51d33b30fce89f2" exitCode=0
Nov 03 20:06:55.325641 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.325508    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" event={"ID":"32c9fcac-59af-4faa-a5ae-0d78688a820b","Type":"ContainerDied","Data":"1592c95b3a604b9241ab73b1405d3ff745ef6025436a5e04f51d33b30fce89f2"}
Nov 03 20:06:55.328806 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.328724    4531 generic.go:356] "Generic (PLEG): container finished" podID="5effd563-1417-489b-bbc4-74a93b08d24d" containerID="2dca25c01e61f165c79beca85476e3d37c1a967f15be6a27a396bf21327e6959" exitCode=0
Nov 03 20:06:55.329086 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.328810    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" event={"ID":"5effd563-1417-489b-bbc4-74a93b08d24d","Type":"ContainerDied","Data":"2dca25c01e61f165c79beca85476e3d37c1a967f15be6a27a396bf21327e6959"}
Nov 03 20:06:55.400041 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.400010    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 20:06:55.435073 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.435045    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 20:06:55.531168 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531134    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-tmp-dir\") pod \"5effd563-1417-489b-bbc4-74a93b08d24d\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") "
Nov 03 20:06:55.531379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531212    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-tmp-dir\") pod \"32c9fcac-59af-4faa-a5ae-0d78688a820b\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") "
Nov 03 20:06:55.531379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531261    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-dshm\") pod \"32c9fcac-59af-4faa-a5ae-0d78688a820b\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") "
Nov 03 20:06:55.531379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531318    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5effd563-1417-489b-bbc4-74a93b08d24d-kube-api-access-kzxhz\" (UniqueName: \"kubernetes.io/projected/5effd563-1417-489b-bbc4-74a93b08d24d-kube-api-access-kzxhz\") pod \"5effd563-1417-489b-bbc4-74a93b08d24d\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") "
Nov 03 20:06:55.531542 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531410    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32c9fcac-59af-4faa-a5ae-0d78688a820b-kube-api-access-v8zwc\" (UniqueName: \"kubernetes.io/projected/32c9fcac-59af-4faa-a5ae-0d78688a820b-kube-api-access-v8zwc\") pod \"32c9fcac-59af-4faa-a5ae-0d78688a820b\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") "
Nov 03 20:06:55.531542 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531411    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-tmp-dir" pod "5effd563-1417-489b-bbc4-74a93b08d24d" (UID: "5effd563-1417-489b-bbc4-74a93b08d24d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.531542 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531455    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-home\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-home\") pod \"32c9fcac-59af-4faa-a5ae-0d78688a820b\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") "
Nov 03 20:06:55.531542 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531484    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-dshm\") pod \"5effd563-1417-489b-bbc4-74a93b08d24d\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") "
Nov 03 20:06:55.531542 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531492    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-tmp-dir" pod "32c9fcac-59af-4faa-a5ae-0d78688a820b" (UID: "32c9fcac-59af-4faa-a5ae-0d78688a820b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.531821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531569    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5effd563-1417-489b-bbc4-74a93b08d24d-tls-certs\" (UniqueName: \"kubernetes.io/secret/5effd563-1417-489b-bbc4-74a93b08d24d-tls-certs\") pod \"5effd563-1417-489b-bbc4-74a93b08d24d\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") "
Nov 03 20:06:55.531821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531610    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-model-cache\") pod \"32c9fcac-59af-4faa-a5ae-0d78688a820b\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") "
Nov 03 20:06:55.531821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531681    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-home\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-home\") pod \"5effd563-1417-489b-bbc4-74a93b08d24d\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") "
Nov 03 20:06:55.531821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531727    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-home" pod "32c9fcac-59af-4faa-a5ae-0d78688a820b" (UID: "32c9fcac-59af-4faa-a5ae-0d78688a820b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.531821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531761    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-model-cache\") pod \"5effd563-1417-489b-bbc4-74a93b08d24d\" (UID: \"5effd563-1417-489b-bbc4-74a93b08d24d\") "
Nov 03 20:06:55.531821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531791    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32c9fcac-59af-4faa-a5ae-0d78688a820b-tls-certs\" (UniqueName: \"kubernetes.io/secret/32c9fcac-59af-4faa-a5ae-0d78688a820b-tls-certs\") pod \"32c9fcac-59af-4faa-a5ae-0d78688a820b\" (UID: \"32c9fcac-59af-4faa-a5ae-0d78688a820b\") "
Nov 03 20:06:55.532271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.531909    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-model-cache" pod "32c9fcac-59af-4faa-a5ae-0d78688a820b" (UID: "32c9fcac-59af-4faa-a5ae-0d78688a820b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.532271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.532086    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-home" pod "5effd563-1417-489b-bbc4-74a93b08d24d" (UID: "5effd563-1417-489b-bbc4-74a93b08d24d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.532271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.532097    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-model-cache" pod "5effd563-1417-489b-bbc4-74a93b08d24d" (UID: "5effd563-1417-489b-bbc4-74a93b08d24d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.532509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.532485    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.532586 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.532538    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.532586 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.532555    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.532586 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.532569    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.532586 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.532582    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.532804 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.532595    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.533864 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.533835    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5effd563-1417-489b-bbc4-74a93b08d24d-kube-api-access-kzxhz" pod "5effd563-1417-489b-bbc4-74a93b08d24d" (UID: "5effd563-1417-489b-bbc4-74a93b08d24d"). InnerVolumeSpecName "kube-api-access-kzxhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:55.533971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.533947    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5effd563-1417-489b-bbc4-74a93b08d24d-tls-certs" pod "5effd563-1417-489b-bbc4-74a93b08d24d" (UID: "5effd563-1417-489b-bbc4-74a93b08d24d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:55.534054 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.534035    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-dshm" pod "5effd563-1417-489b-bbc4-74a93b08d24d" (UID: "5effd563-1417-489b-bbc4-74a93b08d24d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.534108 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.534033    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-dshm" pod "32c9fcac-59af-4faa-a5ae-0d78688a820b" (UID: "32c9fcac-59af-4faa-a5ae-0d78688a820b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.534108 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.534077    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c9fcac-59af-4faa-a5ae-0d78688a820b-kube-api-access-v8zwc" pod "32c9fcac-59af-4faa-a5ae-0d78688a820b" (UID: "32c9fcac-59af-4faa-a5ae-0d78688a820b"). InnerVolumeSpecName "kube-api-access-v8zwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:55.534368 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.534349    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c9fcac-59af-4faa-a5ae-0d78688a820b-tls-certs" pod "32c9fcac-59af-4faa-a5ae-0d78688a820b" (UID: "32c9fcac-59af-4faa-a5ae-0d78688a820b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:55.634360 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.634325    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c9fcac-59af-4faa-a5ae-0d78688a820b-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.634360 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.634357    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c9fcac-59af-4faa-a5ae-0d78688a820b-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.634797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.634370    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzxhz\" (UniqueName: \"kubernetes.io/projected/5effd563-1417-489b-bbc4-74a93b08d24d-kube-api-access-kzxhz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.634797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.634381    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8zwc\" (UniqueName: \"kubernetes.io/projected/32c9fcac-59af-4faa-a5ae-0d78688a820b-kube-api-access-v8zwc\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.634797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.634390    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5effd563-1417-489b-bbc4-74a93b08d24d-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.634797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.634398    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5effd563-1417-489b-bbc4-74a93b08d24d-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.938560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:55.938477    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:56.338832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.338742    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"
Nov 03 20:06:56.338832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.338776    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz" event={"ID":"32c9fcac-59af-4faa-a5ae-0d78688a820b","Type":"ContainerDied","Data":"40fc58f5732278f4fc61a4c9de64c87e5332df0c7c5b12cc7ccc43b4ab2ba3c0"}
Nov 03 20:06:56.338832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.338828    4531 scope.go:122] "RemoveContainer" containerID="1592c95b3a604b9241ab73b1405d3ff745ef6025436a5e04f51d33b30fce89f2"
Nov 03 20:06:56.340600 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.340579    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"
Nov 03 20:06:56.340726 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.340628    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q" event={"ID":"5effd563-1417-489b-bbc4-74a93b08d24d","Type":"ContainerDied","Data":"eba440639f2246bc068b5a7dc2ace4642d92513f0865937d8ffeb5ef6c88ff98"}
Nov 03 20:06:56.349720 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.349700    4531 scope.go:122] "RemoveContainer" containerID="2dca25c01e61f165c79beca85476e3d37c1a967f15be6a27a396bf21327e6959"
Nov 03 20:06:56.362040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.362016    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"]
Nov 03 20:06:56.367716 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.367693    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-w4-f4e123-kserve-6d768c548b-ptfsz"]
Nov 03 20:06:56.375466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.375444    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"]
Nov 03 20:06:56.378402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.378379    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/simulator-w4-f4e123-kserve-55b9dc685d-q942q"]
Nov 03 20:06:56.604229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.604110    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m" podUID="db869375-d3d6-4043-b87f-5df736ed2939" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:06:56.604412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.604261    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94" podUID="752cf009-4d23-4820-ae58-b3ad21317e57" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:06:56.658778 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.642410    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:56.858843 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.858757    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" path="/var/lib/kubelet/pods/32c9fcac-59af-4faa-a5ae-0d78688a820b/volumes"
Nov 03 20:06:56.859341 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:56.859320    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" path="/var/lib/kubelet/pods/5effd563-1417-489b-bbc4-74a93b08d24d/volumes"
Nov 03 20:06:57.546818 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:57.546772    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:58.251197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:58.249759    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:58.958089 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:58.956997    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:59.556211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:59.556162    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:59.727223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:59.724946    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:59.809856 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:59.809775    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"]
Nov 03 20:06:59.814010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:59.813978    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f4e123-j2cqv"]
Nov 03 20:06:59.875112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:06:59.875082    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:00.008983 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:00.008531    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:00.321122 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:00.321083    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:00.814468 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:00.814338    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:00.858663 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:00.858626    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c474d5-b646-4802-acf6-8d57156c1165" path="/var/lib/kubelet/pods/20c474d5-b646-4802-acf6-8d57156c1165/volumes"
Nov 03 20:07:01.521500 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:01.521463    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:02.231658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:02.231553    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:02.839359 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:02.839252    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:03.531560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:03.531522    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:04.134592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:04.130185    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:04.762621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:04.762571    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:05.441644 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:05.441610    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:06.353199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:06.351808    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:06.604407 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:06.604311    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94" podUID="752cf009-4d23-4820-ae58-b3ad21317e57" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:07:06.604578 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:06.604480    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m" podUID="db869375-d3d6-4043-b87f-5df736ed2939" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:07:06.845234 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:06.845163    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:06.978695 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:06.978652    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:07.117752 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:07.117690    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"]
Nov 03 20:07:07.126195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:07.120497    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy" containerID="cri-o://f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee" gracePeriod=30
Nov 03 20:07:07.133466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:07.133435    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:07.135964 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:07.135913    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"]
Nov 03 20:07:07.136264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:07.136243    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy" containerID="cri-o://5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0" gracePeriod=30
Nov 03 20:07:07.317833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:07.317744    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:07.854450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:07.854414    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:08.553193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:08.548828    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:09.084084 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.084042    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"]
Nov 03 20:07:09.084806 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.084782    4531 cpu_manager.go:414] "containerMap: removing container" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main"
Nov 03 20:07:09.084913 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.084808    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main"
Nov 03 20:07:09.084913 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.084833    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7e77260e-a4d1-4097-9859-8ece56379fd0" containerName="revoke-keys"
Nov 03 20:07:09.084913 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.084841    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e77260e-a4d1-4097-9859-8ece56379fd0" containerName="revoke-keys"
Nov 03 20:07:09.084913 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.084873    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main"
Nov 03 20:07:09.084913 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.084880    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main"
Nov 03 20:07:09.084913 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.084893    4531 cpu_manager.go:414] "containerMap: removing container" podUID="75a77b64-e0cd-4766-a0f1-e0b25cf73e54" containerName="revoke-keys"
Nov 03 20:07:09.084913 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.084900    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a77b64-e0cd-4766-a0f1-e0b25cf73e54" containerName="revoke-keys"
Nov 03 20:07:09.085299 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.085052    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="32c9fcac-59af-4faa-a5ae-0d78688a820b" containerName="main"
Nov 03 20:07:09.085299 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.085068    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e77260e-a4d1-4097-9859-8ece56379fd0" containerName="revoke-keys"
Nov 03 20:07:09.085299 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.085082    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5effd563-1417-489b-bbc4-74a93b08d24d" containerName="main"
Nov 03 20:07:09.085299 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.085092    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a77b64-e0cd-4766-a0f1-e0b25cf73e54" containerName="revoke-keys"
Nov 03 20:07:09.090335 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.090242    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:09.093343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.093317    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:07:09.093468 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.093381    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:07:09.094282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.094258    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"]
Nov 03 20:07:09.155677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.155640    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b23a01-7vpsx\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:09.155889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.155685    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gvtx\" (UniqueName: \"kubernetes.io/projected/fd6ddc7e-43da-425e-aca3-22a796701df4-kube-api-access-6gvtx\") pod \"maas-api-revoke-sub-47b23a01-7vpsx\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:09.155889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.155826    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b23a01-7vpsx\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:09.256615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.256578    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b23a01-7vpsx\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:09.256814 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.256627    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gvtx\" (UniqueName: \"kubernetes.io/projected/fd6ddc7e-43da-425e-aca3-22a796701df4-kube-api-access-6gvtx\") pod \"maas-api-revoke-sub-47b23a01-7vpsx\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:09.256814 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.256740    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b23a01-7vpsx\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:09.257344 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.257318    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b23a01-7vpsx\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:09.259097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.259075    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b23a01-7vpsx\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:09.263624 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.263605    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gvtx\" (UniqueName: \"kubernetes.io/projected/fd6ddc7e-43da-425e-aca3-22a796701df4-kube-api-access-6gvtx\") pod \"maas-api-revoke-sub-47b23a01-7vpsx\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:09.376523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.376436    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:09.409301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.409267    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:09.687438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:09.687414    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"]
Nov 03 20:07:09.689597 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:07:09.689554    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6ddc7e_43da_425e_aca3_22a796701df4.slice/crio-30dd6df7903fa0621bae21ecdf01a9bcc9d7e83f85e2247ad107f261c7772631 WatchSource:0}: Error finding container 30dd6df7903fa0621bae21ecdf01a9bcc9d7e83f85e2247ad107f261c7772631: Status 404 returned error can't find the container with id 30dd6df7903fa0621bae21ecdf01a9bcc9d7e83f85e2247ad107f261c7772631
Nov 03 20:07:10.028335 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.028260    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:10.442946 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.442907    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx" event={"ID":"fd6ddc7e-43da-425e-aca3-22a796701df4","Type":"ContainerStarted","Data":"aaac5e997d25e9cbdb781236ba6f40f412238b80f5efcf9d28f212808901b12b"}
Nov 03 20:07:10.443374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.442967    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx" event={"ID":"fd6ddc7e-43da-425e-aca3-22a796701df4","Type":"ContainerStarted","Data":"30dd6df7903fa0621bae21ecdf01a9bcc9d7e83f85e2247ad107f261c7772631"}
Nov 03 20:07:10.458625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.458574    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx" podStartSLOduration=1.458559626 podStartE2EDuration="1.458559626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:10.456066156 +0000 UTC m=+2143.726189450" watchObservedRunningTime="2026-11-03 20:07:10.458559626 +0000 UTC m=+2143.728682985"
Nov 03 20:07:10.719711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.719593    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:10.744155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.743770    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 20:07:10.745981 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.745952    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 20:07:10.776758 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.776716    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db869375-d3d6-4043-b87f-5df736ed2939-kube-api-access-qspsd\" (UniqueName: \"kubernetes.io/projected/db869375-d3d6-4043-b87f-5df736ed2939-kube-api-access-qspsd\") pod \"db869375-d3d6-4043-b87f-5df736ed2939\" (UID: \"db869375-d3d6-4043-b87f-5df736ed2939\") "
Nov 03 20:07:10.776942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.776878    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752cf009-4d23-4820-ae58-b3ad21317e57-kube-api-access-4smrq\" (UniqueName: \"kubernetes.io/projected/752cf009-4d23-4820-ae58-b3ad21317e57-kube-api-access-4smrq\") pod \"752cf009-4d23-4820-ae58-b3ad21317e57\" (UID: \"752cf009-4d23-4820-ae58-b3ad21317e57\") "
Nov 03 20:07:10.776942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.776934    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db869375-d3d6-4043-b87f-5df736ed2939-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db869375-d3d6-4043-b87f-5df736ed2939-plugins-config-volume\") pod \"db869375-d3d6-4043-b87f-5df736ed2939\" (UID: \"db869375-d3d6-4043-b87f-5df736ed2939\") "
Nov 03 20:07:10.777072 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.777034    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752cf009-4d23-4820-ae58-b3ad21317e57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752cf009-4d23-4820-ae58-b3ad21317e57-plugins-config-volume\") pod \"752cf009-4d23-4820-ae58-b3ad21317e57\" (UID: \"752cf009-4d23-4820-ae58-b3ad21317e57\") "
Nov 03 20:07:10.777375 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.777350    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db869375-d3d6-4043-b87f-5df736ed2939-plugins-config-volume" pod "db869375-d3d6-4043-b87f-5df736ed2939" (UID: "db869375-d3d6-4043-b87f-5df736ed2939"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:10.777548 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.777523    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752cf009-4d23-4820-ae58-b3ad21317e57-plugins-config-volume" pod "752cf009-4d23-4820-ae58-b3ad21317e57" (UID: "752cf009-4d23-4820-ae58-b3ad21317e57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:10.777728 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.777705    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db869375-d3d6-4043-b87f-5df736ed2939-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:10.777819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.777734    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752cf009-4d23-4820-ae58-b3ad21317e57-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:10.779498 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.779463    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db869375-d3d6-4043-b87f-5df736ed2939-kube-api-access-qspsd" pod "db869375-d3d6-4043-b87f-5df736ed2939" (UID: "db869375-d3d6-4043-b87f-5df736ed2939"). InnerVolumeSpecName "kube-api-access-qspsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:10.780814 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.780780    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752cf009-4d23-4820-ae58-b3ad21317e57-kube-api-access-4smrq" pod "752cf009-4d23-4820-ae58-b3ad21317e57" (UID: "752cf009-4d23-4820-ae58-b3ad21317e57"). InnerVolumeSpecName "kube-api-access-4smrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:10.878748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.878714    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qspsd\" (UniqueName: \"kubernetes.io/projected/db869375-d3d6-4043-b87f-5df736ed2939-kube-api-access-qspsd\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:10.878748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:10.878751    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4smrq\" (UniqueName: \"kubernetes.io/projected/752cf009-4d23-4820-ae58-b3ad21317e57-kube-api-access-4smrq\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:11.001706 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.001615    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:11.309116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.309085    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:11.451387 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.451351    4531 generic.go:356] "Generic (PLEG): container finished" podID="fd6ddc7e-43da-425e-aca3-22a796701df4" containerID="aaac5e997d25e9cbdb781236ba6f40f412238b80f5efcf9d28f212808901b12b" exitCode=0
Nov 03 20:07:11.451870 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.451438    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx" event={"ID":"fd6ddc7e-43da-425e-aca3-22a796701df4","Type":"ContainerDied","Data":"aaac5e997d25e9cbdb781236ba6f40f412238b80f5efcf9d28f212808901b12b"}
Nov 03 20:07:11.452879 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.452855    4531 generic.go:356] "Generic (PLEG): container finished" podID="752cf009-4d23-4820-ae58-b3ad21317e57" containerID="02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b" exitCode=0
Nov 03 20:07:11.452967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.452885    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94" event={"ID":"752cf009-4d23-4820-ae58-b3ad21317e57","Type":"ContainerDied","Data":"02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b"}
Nov 03 20:07:11.452967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.452897    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"
Nov 03 20:07:11.452967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.452930    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94" event={"ID":"752cf009-4d23-4820-ae58-b3ad21317e57","Type":"ContainerDied","Data":"2f3f17fc22e9ef5d803e68b291007c9ed43d99e6ff18e084ef8091236ffc3a1a"}
Nov 03 20:07:11.452967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.452957    4531 scope.go:122] "RemoveContainer" containerID="02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b"
Nov 03 20:07:11.454597 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.454573    4531 generic.go:356] "Generic (PLEG): container finished" podID="db869375-d3d6-4043-b87f-5df736ed2939" containerID="02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16" exitCode=0
Nov 03 20:07:11.454745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.454618    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"
Nov 03 20:07:11.454860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.454634    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m" event={"ID":"db869375-d3d6-4043-b87f-5df736ed2939","Type":"ContainerDied","Data":"02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16"}
Nov 03 20:07:11.454924 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.454896    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m" event={"ID":"db869375-d3d6-4043-b87f-5df736ed2939","Type":"ContainerDied","Data":"aa337e0db6f82a117e3bf27e49b16ce5fc506764559cc0027606f9dd10bcb134"}
Nov 03 20:07:11.463712 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.463694    4531 scope.go:122] "RemoveContainer" containerID="02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b"
Nov 03 20:07:11.464007 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:11.463988    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b\": container with ID starting with 02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b not found: ID does not exist" containerID="02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b"
Nov 03 20:07:11.464061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.464014    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b"} err="failed to get container status \"02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b\": rpc error: code = NotFound desc = could not find container \"02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b\": container with ID starting with 02003fffa9c2c765a0779bdb5a71323f899ba8e103aafc36976559907500b31b not found: ID does not exist"
Nov 03 20:07:11.464061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.464030    4531 scope.go:122] "RemoveContainer" containerID="02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16"
Nov 03 20:07:11.473925 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.473902    4531 scope.go:122] "RemoveContainer" containerID="02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16"
Nov 03 20:07:11.474269 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:11.474243    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16\": container with ID starting with 02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16 not found: ID does not exist" containerID="02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16"
Nov 03 20:07:11.474361 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.474279    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16"} err="failed to get container status \"02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16\": rpc error: code = NotFound desc = could not find container \"02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16\": container with ID starting with 02248ffa66a00275f747973eada0bb6f8350cda857ea4f22dcd2303ccabdff16 not found: ID does not exist"
Nov 03 20:07:11.481973 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.481951    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"]
Nov 03 20:07:11.484207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.484151    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:07:11.484303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.484221    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:07:11.487515 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.487489    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f4e123-7c57c65699-dbx8m"]
Nov 03 20:07:11.492905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.492883    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"]
Nov 03 20:07:11.496462 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:11.496440    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f4e123-bc4895fdf-kzp94"]
Nov 03 20:07:12.001388 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.001354    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:12.397264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.397239    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 20:07:12.428740 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.428709    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 20:07:12.470986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.470952    4531 generic.go:356] "Generic (PLEG): container finished" podID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerID="f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee" exitCode=0
Nov 03 20:07:12.471514 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.471007    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"
Nov 03 20:07:12.471514 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.471046    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" event={"ID":"fcdaf9c8-7009-4378-8f63-3071e17f5525","Type":"ContainerDied","Data":"f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee"}
Nov 03 20:07:12.471514 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.471117    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm" event={"ID":"fcdaf9c8-7009-4378-8f63-3071e17f5525","Type":"ContainerDied","Data":"7af75fdcd4e86964ad58d0af66bcf439343db668e1aa25635c9280562211f204"}
Nov 03 20:07:12.471514 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.471157    4531 scope.go:122] "RemoveContainer" containerID="f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee"
Nov 03 20:07:12.472802 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.472779    4531 generic.go:356] "Generic (PLEG): container finished" podID="85beffbc-e581-4e37-b139-68118e87212b" containerID="5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0" exitCode=0
Nov 03 20:07:12.472917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.472843    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"
Nov 03 20:07:12.472917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.472895    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" event={"ID":"85beffbc-e581-4e37-b139-68118e87212b","Type":"ContainerDied","Data":"5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0"}
Nov 03 20:07:12.473034 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.472938    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q" event={"ID":"85beffbc-e581-4e37-b139-68118e87212b","Type":"ContainerDied","Data":"47b4e34a4cba4f360f59e4bff1bfcb748a0342a29dd9db7bdb43ea6f556fe336"}
Nov 03 20:07:12.485027 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.485003    4531 scope.go:122] "RemoveContainer" containerID="89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547"
Nov 03 20:07:12.495461 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.495439    4531 scope.go:122] "RemoveContainer" containerID="f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee"
Nov 03 20:07:12.495774 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:12.495752    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee\": container with ID starting with f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee not found: ID does not exist" containerID="f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee"
Nov 03 20:07:12.495876 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.495781    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee"} err="failed to get container status \"f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee\": rpc error: code = NotFound desc = could not find container \"f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee\": container with ID starting with f4241bfe7763c245662ac03181be2a196c90182afffe8658cbe4d8ac099030ee not found: ID does not exist"
Nov 03 20:07:12.495876 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.495799    4531 scope.go:122] "RemoveContainer" containerID="89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547"
Nov 03 20:07:12.496074 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:12.496052    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547\": container with ID starting with 89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547 not found: ID does not exist" containerID="89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547"
Nov 03 20:07:12.496139 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.496082    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547"} err="failed to get container status \"89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547\": rpc error: code = NotFound desc = could not find container \"89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547\": container with ID starting with 89bc69d2cd1b1bd8c385cfd66082d2a7356fc947233be29581f33222ac728547 not found: ID does not exist"
Nov 03 20:07:12.496139 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.496101    4531 scope.go:122] "RemoveContainer" containerID="5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0"
Nov 03 20:07:12.500446 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500420    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-socket\") pod \"85beffbc-e581-4e37-b139-68118e87212b\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") "
Nov 03 20:07:12.500611 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500480    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-certs\") pod \"85beffbc-e581-4e37-b139-68118e87212b\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") "
Nov 03 20:07:12.500611 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500525    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-token\" (UniqueName: \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-token\") pod \"fcdaf9c8-7009-4378-8f63-3071e17f5525\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") "
Nov 03 20:07:12.500611 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500548    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-credential-socket\") pod \"fcdaf9c8-7009-4378-8f63-3071e17f5525\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") "
Nov 03 20:07:12.500611 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500568    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-envoy\") pod \"85beffbc-e581-4e37-b139-68118e87212b\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") "
Nov 03 20:07:12.500611 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500603    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-kube-api-access-m2bcz\" (UniqueName: \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-kube-api-access-m2bcz\") pod \"fcdaf9c8-7009-4378-8f63-3071e17f5525\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") "
Nov 03 20:07:12.500877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500624    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-certs\") pod \"fcdaf9c8-7009-4378-8f63-3071e17f5525\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") "
Nov 03 20:07:12.500877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500684    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-data\") pod \"fcdaf9c8-7009-4378-8f63-3071e17f5525\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") "
Nov 03 20:07:12.500877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500715    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-podinfo\") pod \"fcdaf9c8-7009-4378-8f63-3071e17f5525\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") "
Nov 03 20:07:12.500877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500744    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85beffbc-e581-4e37-b139-68118e87212b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85beffbc-e581-4e37-b139-68118e87212b-istio-podinfo\") pod \"85beffbc-e581-4e37-b139-68118e87212b\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") "
Nov 03 20:07:12.500877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500765    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-socket\") pod \"fcdaf9c8-7009-4378-8f63-3071e17f5525\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") "
Nov 03 20:07:12.500877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500803    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-kube-api-access-4gzpc\" (UniqueName: \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-kube-api-access-4gzpc\") pod \"85beffbc-e581-4e37-b139-68118e87212b\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") "
Nov 03 20:07:12.500877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500833    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-istio-token\" (UniqueName: \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-istio-token\") pod \"85beffbc-e581-4e37-b139-68118e87212b\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") "
Nov 03 20:07:12.500877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500859    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-credential-socket\") pod \"85beffbc-e581-4e37-b139-68118e87212b\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") "
Nov 03 20:07:12.501280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500875    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-credential-socket" pod "fcdaf9c8-7009-4378-8f63-3071e17f5525" (UID: "fcdaf9c8-7009-4378-8f63-3071e17f5525"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.501280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500896    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85beffbc-e581-4e37-b139-68118e87212b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85beffbc-e581-4e37-b139-68118e87212b-istiod-ca-cert\") pod \"85beffbc-e581-4e37-b139-68118e87212b\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") "
Nov 03 20:07:12.501280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500912    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-data\") pod \"85beffbc-e581-4e37-b139-68118e87212b\" (UID: \"85beffbc-e581-4e37-b139-68118e87212b\") "
Nov 03 20:07:12.501280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500939    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-envoy\") pod \"fcdaf9c8-7009-4378-8f63-3071e17f5525\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") "
Nov 03 20:07:12.501280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.500955    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdaf9c8-7009-4378-8f63-3071e17f5525-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdaf9c8-7009-4378-8f63-3071e17f5525-istiod-ca-cert\") pod \"fcdaf9c8-7009-4378-8f63-3071e17f5525\" (UID: \"fcdaf9c8-7009-4378-8f63-3071e17f5525\") "
Nov 03 20:07:12.501280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.501082    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-certs" pod "85beffbc-e581-4e37-b139-68118e87212b" (UID: "85beffbc-e581-4e37-b139-68118e87212b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.501697 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.501546    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.501697 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.501569    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.501697 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.501567    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdaf9c8-7009-4378-8f63-3071e17f5525-istiod-ca-cert" pod "fcdaf9c8-7009-4378-8f63-3071e17f5525" (UID: "fcdaf9c8-7009-4378-8f63-3071e17f5525"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:12.501862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.501680    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-socket" pod "85beffbc-e581-4e37-b139-68118e87212b" (UID: "85beffbc-e581-4e37-b139-68118e87212b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.501928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.501870    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-credential-socket" pod "85beffbc-e581-4e37-b139-68118e87212b" (UID: "85beffbc-e581-4e37-b139-68118e87212b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.502381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.502131    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-socket" pod "fcdaf9c8-7009-4378-8f63-3071e17f5525" (UID: "fcdaf9c8-7009-4378-8f63-3071e17f5525"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.502506 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.502394    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-data" pod "fcdaf9c8-7009-4378-8f63-3071e17f5525" (UID: "fcdaf9c8-7009-4378-8f63-3071e17f5525"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.503235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.502940    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-data" pod "85beffbc-e581-4e37-b139-68118e87212b" (UID: "85beffbc-e581-4e37-b139-68118e87212b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.503235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.503164    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85beffbc-e581-4e37-b139-68118e87212b-istiod-ca-cert" pod "85beffbc-e581-4e37-b139-68118e87212b" (UID: "85beffbc-e581-4e37-b139-68118e87212b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:12.503235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.503203    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-certs" pod "fcdaf9c8-7009-4378-8f63-3071e17f5525" (UID: "fcdaf9c8-7009-4378-8f63-3071e17f5525"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.504453 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.504425    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-istio-token" pod "85beffbc-e581-4e37-b139-68118e87212b" (UID: "85beffbc-e581-4e37-b139-68118e87212b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:12.505491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.505472    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-kube-api-access-m2bcz" pod "fcdaf9c8-7009-4378-8f63-3071e17f5525" (UID: "fcdaf9c8-7009-4378-8f63-3071e17f5525"). InnerVolumeSpecName "kube-api-access-m2bcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:12.506443 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.506401    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-podinfo" pod "fcdaf9c8-7009-4378-8f63-3071e17f5525" (UID: "fcdaf9c8-7009-4378-8f63-3071e17f5525"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:07:12.506643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.506609    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-token" pod "fcdaf9c8-7009-4378-8f63-3071e17f5525" (UID: "fcdaf9c8-7009-4378-8f63-3071e17f5525"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:12.506797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.506773    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-envoy" pod "fcdaf9c8-7009-4378-8f63-3071e17f5525" (UID: "fcdaf9c8-7009-4378-8f63-3071e17f5525"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.506950 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.506926    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-kube-api-access-4gzpc" pod "85beffbc-e581-4e37-b139-68118e87212b" (UID: "85beffbc-e581-4e37-b139-68118e87212b"). InnerVolumeSpecName "kube-api-access-4gzpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:12.507104 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.507076    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-envoy" pod "85beffbc-e581-4e37-b139-68118e87212b" (UID: "85beffbc-e581-4e37-b139-68118e87212b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.508437 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.508410    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85beffbc-e581-4e37-b139-68118e87212b-istio-podinfo" pod "85beffbc-e581-4e37-b139-68118e87212b" (UID: "85beffbc-e581-4e37-b139-68118e87212b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:07:12.510248 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.510124    4531 scope.go:122] "RemoveContainer" containerID="5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0"
Nov 03 20:07:12.510549 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:12.510524    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0\": container with ID starting with 5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0 not found: ID does not exist" containerID="5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0"
Nov 03 20:07:12.510622 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.510562    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0"} err="failed to get container status \"5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0\": rpc error: code = NotFound desc = could not find container \"5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0\": container with ID starting with 5a676ee9f67de4106fbd5c92ef73991e59379f99e11cea7d05b0e1a9165d34d0 not found: ID does not exist"
Nov 03 20:07:12.603118 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603086    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603118 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603116    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdaf9c8-7009-4378-8f63-3071e17f5525-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603118 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603126    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603135    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603149    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603163    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2bcz\" (UniqueName: \"kubernetes.io/projected/fcdaf9c8-7009-4378-8f63-3071e17f5525-kube-api-access-m2bcz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603203    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603214    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603227    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdaf9c8-7009-4378-8f63-3071e17f5525-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603238    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85beffbc-e581-4e37-b139-68118e87212b-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603250    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdaf9c8-7009-4378-8f63-3071e17f5525-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603265    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gzpc\" (UniqueName: \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-kube-api-access-4gzpc\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603277    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85beffbc-e581-4e37-b139-68118e87212b-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603289    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603301    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85beffbc-e581-4e37-b139-68118e87212b-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603323    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85beffbc-e581-4e37-b139-68118e87212b-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.603943 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.603716    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:12.692985 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.692936    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:12.718790 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.708654    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-xks-ca\") pod \"fd6ddc7e-43da-425e-aca3-22a796701df4\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") "
Nov 03 20:07:12.718790 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.708736    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6ddc7e-43da-425e-aca3-22a796701df4-kube-api-access-6gvtx\" (UniqueName: \"kubernetes.io/projected/fd6ddc7e-43da-425e-aca3-22a796701df4-kube-api-access-6gvtx\") pod \"fd6ddc7e-43da-425e-aca3-22a796701df4\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") "
Nov 03 20:07:12.718790 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.708771    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-openshift-ca\") pod \"fd6ddc7e-43da-425e-aca3-22a796701df4\" (UID: \"fd6ddc7e-43da-425e-aca3-22a796701df4\") "
Nov 03 20:07:12.718790 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.711587    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-openshift-ca" pod "fd6ddc7e-43da-425e-aca3-22a796701df4" (UID: "fd6ddc7e-43da-425e-aca3-22a796701df4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:12.732961 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.732226    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6ddc7e-43da-425e-aca3-22a796701df4-kube-api-access-6gvtx" pod "fd6ddc7e-43da-425e-aca3-22a796701df4" (UID: "fd6ddc7e-43da-425e-aca3-22a796701df4"). InnerVolumeSpecName "kube-api-access-6gvtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:12.732961 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.732716    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-xks-ca" pod "fd6ddc7e-43da-425e-aca3-22a796701df4" (UID: "fd6ddc7e-43da-425e-aca3-22a796701df4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:12.797785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.797755    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"]
Nov 03 20:07:12.800642 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.800619    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-m264q"]
Nov 03 20:07:12.808335 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.808309    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"]
Nov 03 20:07:12.810957 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.810934    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.810957 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.810957    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gvtx\" (UniqueName: \"kubernetes.io/projected/fd6ddc7e-43da-425e-aca3-22a796701df4-kube-api-access-6gvtx\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.811118 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.810971    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ddc7e-43da-425e-aca3-22a796701df4-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.812767 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.812747    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-nn8wm"]
Nov 03 20:07:12.860444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.860372    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752cf009-4d23-4820-ae58-b3ad21317e57" path="/var/lib/kubelet/pods/752cf009-4d23-4820-ae58-b3ad21317e57/volumes"
Nov 03 20:07:12.860961 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.860945    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85beffbc-e581-4e37-b139-68118e87212b" path="/var/lib/kubelet/pods/85beffbc-e581-4e37-b139-68118e87212b/volumes"
Nov 03 20:07:12.861393 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.861379    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db869375-d3d6-4043-b87f-5df736ed2939" path="/var/lib/kubelet/pods/db869375-d3d6-4043-b87f-5df736ed2939/volumes"
Nov 03 20:07:12.862212 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:12.862166    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" path="/var/lib/kubelet/pods/fcdaf9c8-7009-4378-8f63-3071e17f5525/volumes"
Nov 03 20:07:13.486168 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:13.486135    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"
Nov 03 20:07:13.486643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:13.486140    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx" event={"ID":"fd6ddc7e-43da-425e-aca3-22a796701df4","Type":"ContainerDied","Data":"30dd6df7903fa0621bae21ecdf01a9bcc9d7e83f85e2247ad107f261c7772631"}
Nov 03 20:07:13.486643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:13.486355    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30dd6df7903fa0621bae21ecdf01a9bcc9d7e83f85e2247ad107f261c7772631"
Nov 03 20:07:13.527962 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:13.527920    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:14.114000 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:14.113962    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:14.906827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:14.906708    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:15.545268 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:15.545225    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:16.208480 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:16.208450    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:16.706361 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:16.706328    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:16.848401 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:16.848353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:17.141961 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:17.141928    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:17.608495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:17.608415    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:18.402121 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:18.402067    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:18.540882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:18.540847    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.148355 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.142082    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.275133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275095    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"]
Nov 03 20:07:19.275618 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275605    4531 cpu_manager.go:414] "containerMap: removing container" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy"
Nov 03 20:07:19.275671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275620    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy"
Nov 03 20:07:19.275671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275641    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fd6ddc7e-43da-425e-aca3-22a796701df4" containerName="revoke-keys"
Nov 03 20:07:19.275671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275647    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6ddc7e-43da-425e-aca3-22a796701df4" containerName="revoke-keys"
Nov 03 20:07:19.275671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275656    4531 cpu_manager.go:414] "containerMap: removing container" podUID="db869375-d3d6-4043-b87f-5df736ed2939" containerName="payload-pre-processing"
Nov 03 20:07:19.275671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275661    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db869375-d3d6-4043-b87f-5df736ed2939" containerName="payload-pre-processing"
Nov 03 20:07:19.275840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275677    4531 cpu_manager.go:414] "containerMap: removing container" podUID="752cf009-4d23-4820-ae58-b3ad21317e57" containerName="payload-processing"
Nov 03 20:07:19.275840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275682    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752cf009-4d23-4820-ae58-b3ad21317e57" containerName="payload-processing"
Nov 03 20:07:19.275840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275700    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy"
Nov 03 20:07:19.275840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275704    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy"
Nov 03 20:07:19.275840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275770    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6ddc7e-43da-425e-aca3-22a796701df4" containerName="revoke-keys"
Nov 03 20:07:19.275840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275780    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="db869375-d3d6-4043-b87f-5df736ed2939" containerName="payload-pre-processing"
Nov 03 20:07:19.275840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275790    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="752cf009-4d23-4820-ae58-b3ad21317e57" containerName="payload-processing"
Nov 03 20:07:19.275840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275799    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy"
Nov 03 20:07:19.275840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275805    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy"
Nov 03 20:07:19.275840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.275810    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="85beffbc-e581-4e37-b139-68118e87212b" containerName="istio-proxy"
Nov 03 20:07:19.280422 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.280401    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:19.283041 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.283012    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:07:19.283401 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.283379    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:07:19.285318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.285293    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"]
Nov 03 20:07:19.413337 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.413227    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e710c56-9tkxf\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:19.413508 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.413346    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqt5j\" (UniqueName: \"kubernetes.io/projected/0d621c61-9042-4275-a9e1-88f2af78f655-kube-api-access-cqt5j\") pod \"maas-api-revoke-sub-9e710c56-9tkxf\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:19.413508 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.413466    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e710c56-9tkxf\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:19.515067 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.515024    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e710c56-9tkxf\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:19.515281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.515083    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqt5j\" (UniqueName: \"kubernetes.io/projected/0d621c61-9042-4275-a9e1-88f2af78f655-kube-api-access-cqt5j\") pod \"maas-api-revoke-sub-9e710c56-9tkxf\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:19.515281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.515137    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e710c56-9tkxf\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:19.515873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.515848    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e710c56-9tkxf\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:19.517570 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.517533    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e710c56-9tkxf\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:19.521898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.521875    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqt5j\" (UniqueName: \"kubernetes.io/projected/0d621c61-9042-4275-a9e1-88f2af78f655-kube-api-access-cqt5j\") pod \"maas-api-revoke-sub-9e710c56-9tkxf\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:19.595708 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.595674    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:19.873803 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.873773    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"]
Nov 03 20:07:19.875552 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:07:19.875515    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d621c61_9042_4275_a9e1_88f2af78f655.slice/crio-e23140490e6e3bc37cfa0fd6dc910b80eec0fda782c92a3c9ae3a509f5b90a2d WatchSource:0}: Error finding container e23140490e6e3bc37cfa0fd6dc910b80eec0fda782c92a3c9ae3a509f5b90a2d: Status 404 returned error can't find the container with id e23140490e6e3bc37cfa0fd6dc910b80eec0fda782c92a3c9ae3a509f5b90a2d
Nov 03 20:07:19.951665 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:19.951458    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:20.537643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:20.537563    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf" event={"ID":"0d621c61-9042-4275-a9e1-88f2af78f655","Type":"ContainerStarted","Data":"77c000d17f4c492a6ee96b1f6a2bfcd3fb79512e91b30226d5e588a9226df233"}
Nov 03 20:07:20.537643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:20.537618    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf" event={"ID":"0d621c61-9042-4275-a9e1-88f2af78f655","Type":"ContainerStarted","Data":"e23140490e6e3bc37cfa0fd6dc910b80eec0fda782c92a3c9ae3a509f5b90a2d"}
Nov 03 20:07:20.552584 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:20.552527    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf" podStartSLOduration=1.552508044 podStartE2EDuration="1.552508044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:20.55125692 +0000 UTC m=+2153.821380216" watchObservedRunningTime="2026-11-03 20:07:20.552508044 +0000 UTC m=+2153.822631338"
Nov 03 20:07:20.573813 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:20.573775    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:07:20.724052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:20.724017    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:21.441295 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:21.441260    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:21.545916 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:21.545879    4531 generic.go:356] "Generic (PLEG): container finished" podID="0d621c61-9042-4275-a9e1-88f2af78f655" containerID="77c000d17f4c492a6ee96b1f6a2bfcd3fb79512e91b30226d5e588a9226df233" exitCode=0
Nov 03 20:07:21.546475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:21.545960    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf" event={"ID":"0d621c61-9042-4275-a9e1-88f2af78f655","Type":"ContainerDied","Data":"77c000d17f4c492a6ee96b1f6a2bfcd3fb79512e91b30226d5e588a9226df233"}
Nov 03 20:07:22.257181 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.257130    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:22.697863 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.697834    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:22.859915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.859841    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d621c61-9042-4275-a9e1-88f2af78f655-kube-api-access-cqt5j\" (UniqueName: \"kubernetes.io/projected/0d621c61-9042-4275-a9e1-88f2af78f655-kube-api-access-cqt5j\") pod \"0d621c61-9042-4275-a9e1-88f2af78f655\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") "
Nov 03 20:07:22.860057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.859985    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-xks-ca\") pod \"0d621c61-9042-4275-a9e1-88f2af78f655\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") "
Nov 03 20:07:22.860097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.860082    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-openshift-ca\") pod \"0d621c61-9042-4275-a9e1-88f2af78f655\" (UID: \"0d621c61-9042-4275-a9e1-88f2af78f655\") "
Nov 03 20:07:22.860558 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.860526    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-openshift-ca" pod "0d621c61-9042-4275-a9e1-88f2af78f655" (UID: "0d621c61-9042-4275-a9e1-88f2af78f655"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:22.862017 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.861989    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-xks-ca" pod "0d621c61-9042-4275-a9e1-88f2af78f655" (UID: "0d621c61-9042-4275-a9e1-88f2af78f655"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:22.862269 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.862111    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d621c61-9042-4275-a9e1-88f2af78f655-kube-api-access-cqt5j" pod "0d621c61-9042-4275-a9e1-88f2af78f655" (UID: "0d621c61-9042-4275-a9e1-88f2af78f655"). InnerVolumeSpecName "kube-api-access-cqt5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:22.938422 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.938214    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:22.961427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.961391    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:22.961427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.961425    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d621c61-9042-4275-a9e1-88f2af78f655-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:22.961427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:22.961436    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqt5j\" (UniqueName: \"kubernetes.io/projected/0d621c61-9042-4275-a9e1-88f2af78f655-kube-api-access-cqt5j\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:23.569138 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:23.569103    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"
Nov 03 20:07:23.569396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:23.569138    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf" event={"ID":"0d621c61-9042-4275-a9e1-88f2af78f655","Type":"ContainerDied","Data":"e23140490e6e3bc37cfa0fd6dc910b80eec0fda782c92a3c9ae3a509f5b90a2d"}
Nov 03 20:07:23.569396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:23.569202    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e23140490e6e3bc37cfa0fd6dc910b80eec0fda782c92a3c9ae3a509f5b90a2d"
Nov 03 20:07:23.872674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:23.872554    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:24.560978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:24.560942    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:24.717266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:24.717223    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:25.430344 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.426884    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:25.639640 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.639606    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"]
Nov 03 20:07:25.640096 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.640084    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy"
Nov 03 20:07:25.640135 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.640098    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdaf9c8-7009-4378-8f63-3071e17f5525" containerName="istio-proxy"
Nov 03 20:07:25.640135 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.640119    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0d621c61-9042-4275-a9e1-88f2af78f655" containerName="revoke-keys"
Nov 03 20:07:25.640135 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.640125    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d621c61-9042-4275-a9e1-88f2af78f655" containerName="revoke-keys"
Nov 03 20:07:25.640282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.640238    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d621c61-9042-4275-a9e1-88f2af78f655" containerName="revoke-keys"
Nov 03 20:07:25.643698 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.643680    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:25.646850 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.646822    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:07:25.646979 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.646898    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:07:25.649521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.649498    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"]
Nov 03 20:07:25.734413 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.734328    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785aeb92-tkc7b\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:25.734413 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.734394    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785aeb92-tkc7b\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:25.734613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.734448    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rcs\" (UniqueName: \"kubernetes.io/projected/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-kube-api-access-68rcs\") pod \"maas-api-revoke-sub-785aeb92-tkc7b\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:25.835661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.835624    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785aeb92-tkc7b\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:25.835850 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.835694    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785aeb92-tkc7b\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:25.835850 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.835713    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rcs\" (UniqueName: \"kubernetes.io/projected/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-kube-api-access-68rcs\") pod \"maas-api-revoke-sub-785aeb92-tkc7b\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:25.836320 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.836291    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785aeb92-tkc7b\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:25.838117 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.838080    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785aeb92-tkc7b\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:25.842923 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.842902    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rcs\" (UniqueName: \"kubernetes.io/projected/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-kube-api-access-68rcs\") pod \"maas-api-revoke-sub-785aeb92-tkc7b\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:25.961431 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:25.961396    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:26.145055 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:26.145017    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:26.261525 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:26.253273    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"]
Nov 03 20:07:26.591746 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:26.591711    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b" event={"ID":"d1441a6f-426c-4869-a3da-b0b0be8c0d6d","Type":"ContainerStarted","Data":"805eae836ce462caff3d5bac2d9bce3477ac3c29cffb470f552d80e1dea14551"}
Nov 03 20:07:26.592196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:26.591777    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b" event={"ID":"d1441a6f-426c-4869-a3da-b0b0be8c0d6d","Type":"ContainerStarted","Data":"c2adca41911bb1f9c912df12a42e9c0046846c59bc1b1b1dab39ca61fcbeeaf7"}
Nov 03 20:07:26.614021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:26.613958    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b" podStartSLOduration=1.613939276 podStartE2EDuration="1.613939276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:26.611752175 +0000 UTC m=+2159.881875472" watchObservedRunningTime="2026-11-03 20:07:26.613939276 +0000 UTC m=+2159.884062570"
Nov 03 20:07:26.752937 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:26.752899    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:27.429673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:27.429465    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:27.599661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:27.599624    4531 generic.go:356] "Generic (PLEG): container finished" podID="d1441a6f-426c-4869-a3da-b0b0be8c0d6d" containerID="805eae836ce462caff3d5bac2d9bce3477ac3c29cffb470f552d80e1dea14551" exitCode=0
Nov 03 20:07:27.600110 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:27.599716    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b" event={"ID":"d1441a6f-426c-4869-a3da-b0b0be8c0d6d","Type":"ContainerDied","Data":"805eae836ce462caff3d5bac2d9bce3477ac3c29cffb470f552d80e1dea14551"}
Nov 03 20:07:28.161075 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.161011    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:28.759817 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.759786    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:28.794070 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.794025    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-xks-ca\") pod \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") "
Nov 03 20:07:28.794304 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.794086    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-kube-api-access-68rcs\" (UniqueName: \"kubernetes.io/projected/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-kube-api-access-68rcs\") pod \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") "
Nov 03 20:07:28.794304 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.794139    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-openshift-ca\") pod \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\" (UID: \"d1441a6f-426c-4869-a3da-b0b0be8c0d6d\") "
Nov 03 20:07:28.794724 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.794666    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-openshift-ca" pod "d1441a6f-426c-4869-a3da-b0b0be8c0d6d" (UID: "d1441a6f-426c-4869-a3da-b0b0be8c0d6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:28.795483 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.795454    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:28.796667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.796643    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-xks-ca" pod "d1441a6f-426c-4869-a3da-b0b0be8c0d6d" (UID: "d1441a6f-426c-4869-a3da-b0b0be8c0d6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:28.797044 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.797024    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-kube-api-access-68rcs" pod "d1441a6f-426c-4869-a3da-b0b0be8c0d6d" (UID: "d1441a6f-426c-4869-a3da-b0b0be8c0d6d"). InnerVolumeSpecName "kube-api-access-68rcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:28.894766 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.894685    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:28.898154 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.898090    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:28.898154 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:28.898126    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68rcs\" (UniqueName: \"kubernetes.io/projected/d1441a6f-426c-4869-a3da-b0b0be8c0d6d-kube-api-access-68rcs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:29.640522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:29.640467    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b" event={"ID":"d1441a6f-426c-4869-a3da-b0b0be8c0d6d","Type":"ContainerDied","Data":"c2adca41911bb1f9c912df12a42e9c0046846c59bc1b1b1dab39ca61fcbeeaf7"}
Nov 03 20:07:29.640699 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:29.640521    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"
Nov 03 20:07:29.640699 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:29.640544    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2adca41911bb1f9c912df12a42e9c0046846c59bc1b1b1dab39ca61fcbeeaf7"
Nov 03 20:07:29.651458 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:29.651432    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:30.348685 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:30.344377    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:30.985380 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:30.985338    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"]
Nov 03 20:07:31.039901 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:31.039866    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:31.176745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:31.176708    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:31.833688 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:31.833646    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:31.837082 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:31.837051    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"]
Nov 03 20:07:31.966949 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:31.966896    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:32.002873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.002839    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"]
Nov 03 20:07:32.003542 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.003522    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d1441a6f-426c-4869-a3da-b0b0be8c0d6d" containerName="revoke-keys"
Nov 03 20:07:32.003619 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.003542    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1441a6f-426c-4869-a3da-b0b0be8c0d6d" containerName="revoke-keys"
Nov 03 20:07:32.003677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.003655    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1441a6f-426c-4869-a3da-b0b0be8c0d6d" containerName="revoke-keys"
Nov 03 20:07:32.007212 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.007193    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:32.010787 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.010758    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:07:32.011294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.011274    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:07:32.015222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.014993    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"]
Nov 03 20:07:32.142690 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.142642    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:32.142690 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.142689    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:32.142995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.142807    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbdgf\" (UniqueName: \"kubernetes.io/projected/0f03d949-2ae9-46e1-9cf7-44b06607d08a-kube-api-access-nbdgf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:32.244557 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.244520    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:32.244751 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.244564    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:32.244751 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.244609    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbdgf\" (UniqueName: \"kubernetes.io/projected/0f03d949-2ae9-46e1-9cf7-44b06607d08a-kube-api-access-nbdgf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:32.245448 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.245422    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:32.247088 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.247060    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:32.251310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.251289    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbdgf\" (UniqueName: \"kubernetes.io/projected/0f03d949-2ae9-46e1-9cf7-44b06607d08a-kube-api-access-nbdgf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:32.335159 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.335116    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:32.613783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.613740    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"]
Nov 03 20:07:32.614017 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:07:32.613991    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f03d949_2ae9_46e1_9cf7_44b06607d08a.slice/crio-4756ca5f3f69034227469455d4a81c147df49c0e347756a974ca8c32d5ebe850 WatchSource:0}: Error finding container 4756ca5f3f69034227469455d4a81c147df49c0e347756a974ca8c32d5ebe850: Status 404 returned error can't find the container with id 4756ca5f3f69034227469455d4a81c147df49c0e347756a974ca8c32d5ebe850
Nov 03 20:07:32.662862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.662827    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd" event={"ID":"0f03d949-2ae9-46e1-9cf7-44b06607d08a","Type":"ContainerStarted","Data":"4756ca5f3f69034227469455d4a81c147df49c0e347756a974ca8c32d5ebe850"}
Nov 03 20:07:32.790288 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:32.789907    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:33.522799 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:33.522765    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:33.671448 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:33.671410    4531 generic.go:356] "Generic (PLEG): container finished" podID="0f03d949-2ae9-46e1-9cf7-44b06607d08a" containerID="e6df4de8909aecc1169ebd66717384f15ffd66831abb09c6943f2dacb65683be" exitCode=0
Nov 03 20:07:33.671666 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:33.671494    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd" event={"ID":"0f03d949-2ae9-46e1-9cf7-44b06607d08a","Type":"ContainerDied","Data":"e6df4de8909aecc1169ebd66717384f15ffd66831abb09c6943f2dacb65683be"}
Nov 03 20:07:34.144349 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.144313    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:34.798775 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.798727    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:34.839021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.838993    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:34.872935 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.872899    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f03d949-2ae9-46e1-9cf7-44b06607d08a-kube-api-access-nbdgf\" (UniqueName: \"kubernetes.io/projected/0f03d949-2ae9-46e1-9cf7-44b06607d08a-kube-api-access-nbdgf\") pod \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") "
Nov 03 20:07:34.873134 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.873036    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-xks-ca\") pod \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") "
Nov 03 20:07:34.873301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.873205    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-openshift-ca\") pod \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\" (UID: \"0f03d949-2ae9-46e1-9cf7-44b06607d08a\") "
Nov 03 20:07:34.873613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.873588    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-openshift-ca" pod "0f03d949-2ae9-46e1-9cf7-44b06607d08a" (UID: "0f03d949-2ae9-46e1-9cf7-44b06607d08a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:34.873717 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.873700    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:34.875305 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.875280    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-xks-ca" pod "0f03d949-2ae9-46e1-9cf7-44b06607d08a" (UID: "0f03d949-2ae9-46e1-9cf7-44b06607d08a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:34.877833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.877797    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f03d949-2ae9-46e1-9cf7-44b06607d08a-kube-api-access-nbdgf" pod "0f03d949-2ae9-46e1-9cf7-44b06607d08a" (UID: "0f03d949-2ae9-46e1-9cf7-44b06607d08a"). InnerVolumeSpecName "kube-api-access-nbdgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:34.975573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.975490    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbdgf\" (UniqueName: \"kubernetes.io/projected/0f03d949-2ae9-46e1-9cf7-44b06607d08a-kube-api-access-nbdgf\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:34.975573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:34.975523    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f03d949-2ae9-46e1-9cf7-44b06607d08a-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:35.424639 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.416623    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:35.532606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.532572    4531 scope.go:122] "RemoveContainer" containerID="62e35dc414d3af4d379a31079527879cdd210b7d3df21d2e10e09adf3f3ab0d8"
Nov 03 20:07:35.687494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.687403    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"
Nov 03 20:07:35.687494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.687422    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd" event={"ID":"0f03d949-2ae9-46e1-9cf7-44b06607d08a","Type":"ContainerDied","Data":"4756ca5f3f69034227469455d4a81c147df49c0e347756a974ca8c32d5ebe850"}
Nov 03 20:07:35.687494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.687480    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4756ca5f3f69034227469455d4a81c147df49c0e347756a974ca8c32d5ebe850"
Nov 03 20:07:35.841124 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.841083    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"]
Nov 03 20:07:35.841827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.841803    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0f03d949-2ae9-46e1-9cf7-44b06607d08a" containerName="revoke-keys"
Nov 03 20:07:35.841992 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.841828    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f03d949-2ae9-46e1-9cf7-44b06607d08a" containerName="revoke-keys"
Nov 03 20:07:35.842068 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.842037    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f03d949-2ae9-46e1-9cf7-44b06607d08a" containerName="revoke-keys"
Nov 03 20:07:35.847322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.847295    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:35.850904 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.850875    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:07:35.850904 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.850910    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:07:35.851476 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.851454    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"]
Nov 03 20:07:35.886182 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.886129    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c812237a-fwlxz\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:35.886454 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.886431    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c812237a-fwlxz\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:35.886559 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.886543    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnf2g\" (UniqueName: \"kubernetes.io/projected/8d1f628d-2a51-4bf7-a162-a800effecda7-kube-api-access-mnf2g\") pod \"maas-api-revoke-sub-c812237a-fwlxz\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:35.987482 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.987393    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c812237a-fwlxz\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:35.987668 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.987486    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnf2g\" (UniqueName: \"kubernetes.io/projected/8d1f628d-2a51-4bf7-a162-a800effecda7-kube-api-access-mnf2g\") pod \"maas-api-revoke-sub-c812237a-fwlxz\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:35.987668 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.987532    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c812237a-fwlxz\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:35.988214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.988191    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c812237a-fwlxz\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:35.989753 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.989731    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c812237a-fwlxz\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:35.994598 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:35.994569    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnf2g\" (UniqueName: \"kubernetes.io/projected/8d1f628d-2a51-4bf7-a162-a800effecda7-kube-api-access-mnf2g\") pod \"maas-api-revoke-sub-c812237a-fwlxz\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:36.165790 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:36.165742    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:36.236227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:36.232502    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:36.515335 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:36.515302    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"]
Nov 03 20:07:36.518151 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:07:36.518117    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1f628d_2a51_4bf7_a162_a800effecda7.slice/crio-a665b90be27ce722797709d1fe9e95b0af8e7cdd4b45cc481f426e111bf051e9 WatchSource:0}: Error finding container a665b90be27ce722797709d1fe9e95b0af8e7cdd4b45cc481f426e111bf051e9: Status 404 returned error can't find the container with id a665b90be27ce722797709d1fe9e95b0af8e7cdd4b45cc481f426e111bf051e9
Nov 03 20:07:36.699994 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:36.699941    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz" event={"ID":"8d1f628d-2a51-4bf7-a162-a800effecda7","Type":"ContainerStarted","Data":"a665b90be27ce722797709d1fe9e95b0af8e7cdd4b45cc481f426e111bf051e9"}
Nov 03 20:07:37.022641 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:37.020080    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:37.653768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:37.649882    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:37.709153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:37.709118    4531 generic.go:356] "Generic (PLEG): container finished" podID="8d1f628d-2a51-4bf7-a162-a800effecda7" containerID="48e5ce457e0d9a86ab0862649d3283376360d609593c5104b9066bff8f0fde8f" exitCode=0
Nov 03 20:07:37.709372 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:37.709220    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz" event={"ID":"8d1f628d-2a51-4bf7-a162-a800effecda7","Type":"ContainerDied","Data":"48e5ce457e0d9a86ab0862649d3283376360d609593c5104b9066bff8f0fde8f"}
Nov 03 20:07:38.362023 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:38.360487    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:38.873556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:38.873516    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:38.930574 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:38.930533    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-openshift-ca\") pod \"8d1f628d-2a51-4bf7-a162-a800effecda7\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") "
Nov 03 20:07:38.930806 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:38.930610    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-xks-ca\") pod \"8d1f628d-2a51-4bf7-a162-a800effecda7\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") "
Nov 03 20:07:38.930806 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:38.930663    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1f628d-2a51-4bf7-a162-a800effecda7-kube-api-access-mnf2g\" (UniqueName: \"kubernetes.io/projected/8d1f628d-2a51-4bf7-a162-a800effecda7-kube-api-access-mnf2g\") pod \"8d1f628d-2a51-4bf7-a162-a800effecda7\" (UID: \"8d1f628d-2a51-4bf7-a162-a800effecda7\") "
Nov 03 20:07:38.931111 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:38.931082    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-openshift-ca" pod "8d1f628d-2a51-4bf7-a162-a800effecda7" (UID: "8d1f628d-2a51-4bf7-a162-a800effecda7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:38.931344 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:38.931314    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:38.932969 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:38.932943    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-xks-ca" pod "8d1f628d-2a51-4bf7-a162-a800effecda7" (UID: "8d1f628d-2a51-4bf7-a162-a800effecda7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:38.933056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:38.932971    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1f628d-2a51-4bf7-a162-a800effecda7-kube-api-access-mnf2g" pod "8d1f628d-2a51-4bf7-a162-a800effecda7" (UID: "8d1f628d-2a51-4bf7-a162-a800effecda7"). InnerVolumeSpecName "kube-api-access-mnf2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:38.997237 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:38.997034    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:39.032883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:39.032846    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1f628d-2a51-4bf7-a162-a800effecda7-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:39.032883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:39.032882    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnf2g\" (UniqueName: \"kubernetes.io/projected/8d1f628d-2a51-4bf7-a162-a800effecda7-kube-api-access-mnf2g\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:39.638788 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:39.636310    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:39.724220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:39.724169    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"
Nov 03 20:07:39.724429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:39.724233    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz" event={"ID":"8d1f628d-2a51-4bf7-a162-a800effecda7","Type":"ContainerDied","Data":"a665b90be27ce722797709d1fe9e95b0af8e7cdd4b45cc481f426e111bf051e9"}
Nov 03 20:07:39.724429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:39.724287    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a665b90be27ce722797709d1fe9e95b0af8e7cdd4b45cc481f426e111bf051e9"
Nov 03 20:07:40.088610 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:40.088515    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"]
Nov 03 20:07:40.099191 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:40.099147    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc16508-hdxqj"]
Nov 03 20:07:40.330244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:40.330146    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:40.858854 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:40.858822    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc17f74e-f63e-4614-9926-be37b19614a3" path="/var/lib/kubelet/pods/dc17f74e-f63e-4614-9926-be37b19614a3/volumes"
Nov 03 20:07:40.962077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:40.961931    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:41.644581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:41.644479    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:42.131986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:42.131944    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"]
Nov 03 20:07:42.143670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:42.143632    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"]
Nov 03 20:07:42.262828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:42.262767    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:42.408951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:42.408849    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:42.968754 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:42.968720    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:43.150948 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:43.150909    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:43.843205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:43.841863    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:44.516137 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:44.516098    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:45.229287 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:45.229257    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:45.808031 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:45.807998    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:45.946741 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:45.946708    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:46.016647 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.016606    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"]
Nov 03 20:07:46.024569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.024532    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs" podUID="a841ef7e-4a8d-469b-b13e-4d84980e3b1c" containerName="main" containerID="cri-o://2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31" gracePeriod=45
Nov 03 20:07:46.043531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.043491    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3ff278-2sjjd"]
Nov 03 20:07:46.083924 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.083836    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:46.220459 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.220339    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:46.349097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.349066    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:07:46.462977 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.462942    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-kube-api-access-l8wkz\" (UniqueName: \"kubernetes.io/projected/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-kube-api-access-l8wkz\") pod \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") "
Nov 03 20:07:46.463209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.463005    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-model-cache\") pod \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") "
Nov 03 20:07:46.463209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.463106    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tls-certs\") pod \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") "
Nov 03 20:07:46.463304 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.463269    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-dshm\") pod \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") "
Nov 03 20:07:46.463362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.463318    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-model-cache" pod "a841ef7e-4a8d-469b-b13e-4d84980e3b1c" (UID: "a841ef7e-4a8d-469b-b13e-4d84980e3b1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:46.463362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.463326    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-home\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-home\") pod \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") "
Nov 03 20:07:46.463450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.463422    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tmp-dir\") pod \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\" (UID: \"a841ef7e-4a8d-469b-b13e-4d84980e3b1c\") "
Nov 03 20:07:46.463567 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.463544    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-home" pod "a841ef7e-4a8d-469b-b13e-4d84980e3b1c" (UID: "a841ef7e-4a8d-469b-b13e-4d84980e3b1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:46.463744 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.463714    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tmp-dir" pod "a841ef7e-4a8d-469b-b13e-4d84980e3b1c" (UID: "a841ef7e-4a8d-469b-b13e-4d84980e3b1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:46.463927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.463910    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:46.464000 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.463933    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:46.464000 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.463945    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:46.465314 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.465291    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-dshm" pod "a841ef7e-4a8d-469b-b13e-4d84980e3b1c" (UID: "a841ef7e-4a8d-469b-b13e-4d84980e3b1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:46.465430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.465363    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tls-certs" pod "a841ef7e-4a8d-469b-b13e-4d84980e3b1c" (UID: "a841ef7e-4a8d-469b-b13e-4d84980e3b1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:46.465479 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.465436    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-kube-api-access-l8wkz" pod "a841ef7e-4a8d-469b-b13e-4d84980e3b1c" (UID: "a841ef7e-4a8d-469b-b13e-4d84980e3b1c"). InnerVolumeSpecName "kube-api-access-l8wkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:46.565728 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.565691    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:46.565728 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.565723    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8wkz\" (UniqueName: \"kubernetes.io/projected/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-kube-api-access-l8wkz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:46.565970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.565741    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a841ef7e-4a8d-469b-b13e-4d84980e3b1c-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:46.579494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.579452    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:46.778345 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.778306    4531 generic.go:356] "Generic (PLEG): container finished" podID="a841ef7e-4a8d-469b-b13e-4d84980e3b1c" containerID="2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31" exitCode=0
Nov 03 20:07:46.778569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.778356    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"
Nov 03 20:07:46.778569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.778364    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs" event={"ID":"a841ef7e-4a8d-469b-b13e-4d84980e3b1c","Type":"ContainerDied","Data":"2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31"}
Nov 03 20:07:46.778569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.778406    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs" event={"ID":"a841ef7e-4a8d-469b-b13e-4d84980e3b1c","Type":"ContainerDied","Data":"d530f0759a432aff2b74374fc3201e8bba62e9f78722a31c2555aaa9b4952f0b"}
Nov 03 20:07:46.778569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.778423    4531 scope.go:122] "RemoveContainer" containerID="2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31"
Nov 03 20:07:46.792101 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.792082    4531 scope.go:122] "RemoveContainer" containerID="2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31"
Nov 03 20:07:46.792426 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:46.792403    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31\": container with ID starting with 2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31 not found: ID does not exist" containerID="2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31"
Nov 03 20:07:46.792492 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.792438    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31"} err="failed to get container status \"2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31\": rpc error: code = NotFound desc = could not find container \"2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31\": container with ID starting with 2d571131c6f600e9048882433daa0c7dcc43f85ca94f1913aac491461bcf5d31 not found: ID does not exist"
Nov 03 20:07:46.803885 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.803855    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"]
Nov 03 20:07:46.807335 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.807312    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs"]
Nov 03 20:07:46.858744 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.858712    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f03d949-2ae9-46e1-9cf7-44b06607d08a" path="/var/lib/kubelet/pods/0f03d949-2ae9-46e1-9cf7-44b06607d08a/volumes"
Nov 03 20:07:46.859400 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.859378    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a841ef7e-4a8d-469b-b13e-4d84980e3b1c" path="/var/lib/kubelet/pods/a841ef7e-4a8d-469b-b13e-4d84980e3b1c/volumes"
Nov 03 20:07:46.861842 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.861806    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c" podUID="8545f1c7-81a0-42a1-871e-47ca9a231c2d" containerName="main" containerID="cri-o://43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f" gracePeriod=45
Nov 03 20:07:46.946090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:46.946023    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:47.133293 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.133246    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h" podUID="7987b86a-8744-4e09-a949-c23671975702" containerName="payload-processing" containerID="cri-o://3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f" gracePeriod=25
Nov 03 20:07:47.144964 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.144928    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd" podUID="0b7c7784-0528-47bd-b6a8-b42778e96d58" containerName="payload-pre-processing" containerID="cri-o://1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6" gracePeriod=25
Nov 03 20:07:47.296657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.296577    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:07:47.312233 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.311856    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-3ff278/auth-test-model-3ff278-kserve-5899d7476f-7pcxs" podUID="a841ef7e-4a8d-469b-b13e-4d84980e3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.238:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:47.377932 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.377893    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-model-cache\") pod \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") "
Nov 03 20:07:47.378147 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.377971    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-dshm\") pod \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") "
Nov 03 20:07:47.378310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.378289    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-model-cache" pod "8545f1c7-81a0-42a1-871e-47ca9a231c2d" (UID: "8545f1c7-81a0-42a1-871e-47ca9a231c2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:47.378397 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.378338    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8545f1c7-81a0-42a1-871e-47ca9a231c2d-kube-api-access-jw6tk\" (UniqueName: \"kubernetes.io/projected/8545f1c7-81a0-42a1-871e-47ca9a231c2d-kube-api-access-jw6tk\") pod \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") "
Nov 03 20:07:47.378397 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.378386    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-home\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-home\") pod \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") "
Nov 03 20:07:47.378495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.378429    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tls-certs\" (UniqueName: \"kubernetes.io/secret/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tls-certs\") pod \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") "
Nov 03 20:07:47.378495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.378453    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tmp-dir\") pod \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\" (UID: \"8545f1c7-81a0-42a1-871e-47ca9a231c2d\") "
Nov 03 20:07:47.378844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.378823    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:47.379067 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.379044    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tmp-dir" pod "8545f1c7-81a0-42a1-871e-47ca9a231c2d" (UID: "8545f1c7-81a0-42a1-871e-47ca9a231c2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:47.379300 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.379279    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-home" pod "8545f1c7-81a0-42a1-871e-47ca9a231c2d" (UID: "8545f1c7-81a0-42a1-871e-47ca9a231c2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:47.382061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.382031    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tls-certs" pod "8545f1c7-81a0-42a1-871e-47ca9a231c2d" (UID: "8545f1c7-81a0-42a1-871e-47ca9a231c2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:47.382151 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.382111    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-dshm" pod "8545f1c7-81a0-42a1-871e-47ca9a231c2d" (UID: "8545f1c7-81a0-42a1-871e-47ca9a231c2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:47.383554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.383535    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8545f1c7-81a0-42a1-871e-47ca9a231c2d-kube-api-access-jw6tk" pod "8545f1c7-81a0-42a1-871e-47ca9a231c2d" (UID: "8545f1c7-81a0-42a1-871e-47ca9a231c2d"). InnerVolumeSpecName "kube-api-access-jw6tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:47.454708 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.454677    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:07:47.469065 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.469038    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:07:47.479608 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.479578    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7c7784-0528-47bd-b6a8-b42778e96d58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7c7784-0528-47bd-b6a8-b42778e96d58-plugins-config-volume\") pod \"0b7c7784-0528-47bd-b6a8-b42778e96d58\" (UID: \"0b7c7784-0528-47bd-b6a8-b42778e96d58\") "
Nov 03 20:07:47.479771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.479631    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7987b86a-8744-4e09-a949-c23671975702-kube-api-access-89x55\" (UniqueName: \"kubernetes.io/projected/7987b86a-8744-4e09-a949-c23671975702-kube-api-access-89x55\") pod \"7987b86a-8744-4e09-a949-c23671975702\" (UID: \"7987b86a-8744-4e09-a949-c23671975702\") "
Nov 03 20:07:47.479771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.479748    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7987b86a-8744-4e09-a949-c23671975702-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7987b86a-8744-4e09-a949-c23671975702-plugins-config-volume\") pod \"7987b86a-8744-4e09-a949-c23671975702\" (UID: \"7987b86a-8744-4e09-a949-c23671975702\") "
Nov 03 20:07:47.479907 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.479801    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7c7784-0528-47bd-b6a8-b42778e96d58-kube-api-access-2r74j\" (UniqueName: \"kubernetes.io/projected/0b7c7784-0528-47bd-b6a8-b42778e96d58-kube-api-access-2r74j\") pod \"0b7c7784-0528-47bd-b6a8-b42778e96d58\" (UID: \"0b7c7784-0528-47bd-b6a8-b42778e96d58\") "
Nov 03 20:07:47.480019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.479991    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7c7784-0528-47bd-b6a8-b42778e96d58-plugins-config-volume" pod "0b7c7784-0528-47bd-b6a8-b42778e96d58" (UID: "0b7c7784-0528-47bd-b6a8-b42778e96d58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:47.480130 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.480110    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7987b86a-8744-4e09-a949-c23671975702-plugins-config-volume" pod "7987b86a-8744-4e09-a949-c23671975702" (UID: "7987b86a-8744-4e09-a949-c23671975702"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:47.480270 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.480253    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7c7784-0528-47bd-b6a8-b42778e96d58-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:47.480321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.480272    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw6tk\" (UniqueName: \"kubernetes.io/projected/8545f1c7-81a0-42a1-871e-47ca9a231c2d-kube-api-access-jw6tk\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:47.480321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.480282    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:47.480321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.480292    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:47.480321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.480300    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:47.480321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.480308    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7987b86a-8744-4e09-a949-c23671975702-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:47.480321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.480316    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8545f1c7-81a0-42a1-871e-47ca9a231c2d-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:47.481874 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.481857    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7987b86a-8744-4e09-a949-c23671975702-kube-api-access-89x55" pod "7987b86a-8744-4e09-a949-c23671975702" (UID: "7987b86a-8744-4e09-a949-c23671975702"). InnerVolumeSpecName "kube-api-access-89x55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:47.481940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.481874    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7c7784-0528-47bd-b6a8-b42778e96d58-kube-api-access-2r74j" pod "0b7c7784-0528-47bd-b6a8-b42778e96d58" (UID: "0b7c7784-0528-47bd-b6a8-b42778e96d58"). InnerVolumeSpecName "kube-api-access-2r74j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:47.581926 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.581821    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r74j\" (UniqueName: \"kubernetes.io/projected/0b7c7784-0528-47bd-b6a8-b42778e96d58-kube-api-access-2r74j\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:47.581926 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.581867    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89x55\" (UniqueName: \"kubernetes.io/projected/7987b86a-8744-4e09-a949-c23671975702-kube-api-access-89x55\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:47.725959 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.725928    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:47.804199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.804133    4531 generic.go:356] "Generic (PLEG): container finished" podID="7987b86a-8744-4e09-a949-c23671975702" containerID="3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f" exitCode=0
Nov 03 20:07:47.804399 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.804348    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h" event={"ID":"7987b86a-8744-4e09-a949-c23671975702","Type":"ContainerDied","Data":"3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f"}
Nov 03 20:07:47.804504 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.804446    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h" event={"ID":"7987b86a-8744-4e09-a949-c23671975702","Type":"ContainerDied","Data":"180828f720f68e57b1e295141f034eaf50590789e6ffea2c7f4b58536c039cd7"}
Nov 03 20:07:47.804504 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.804477    4531 scope.go:122] "RemoveContainer" containerID="3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f"
Nov 03 20:07:47.804671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.804653    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"
Nov 03 20:07:47.809491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.809463    4531 generic.go:356] "Generic (PLEG): container finished" podID="8545f1c7-81a0-42a1-871e-47ca9a231c2d" containerID="43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f" exitCode=0
Nov 03 20:07:47.809653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.809574    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c" event={"ID":"8545f1c7-81a0-42a1-871e-47ca9a231c2d","Type":"ContainerDied","Data":"43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f"}
Nov 03 20:07:47.809653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.809608    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c" event={"ID":"8545f1c7-81a0-42a1-871e-47ca9a231c2d","Type":"ContainerDied","Data":"86e0d41fcd52283d6b860691ca47b758b385b6a4c434122517669c3e105fe42e"}
Nov 03 20:07:47.809771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.809670    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"
Nov 03 20:07:47.819954 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.819920    4531 generic.go:356] "Generic (PLEG): container finished" podID="0b7c7784-0528-47bd-b6a8-b42778e96d58" containerID="1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6" exitCode=0
Nov 03 20:07:47.820112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.820051    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"
Nov 03 20:07:47.820946 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.820923    4531 scope.go:122] "RemoveContainer" containerID="3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f"
Nov 03 20:07:47.821892 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.821801    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd" event={"ID":"0b7c7784-0528-47bd-b6a8-b42778e96d58","Type":"ContainerDied","Data":"1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6"}
Nov 03 20:07:47.821892 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.821857    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd" event={"ID":"0b7c7784-0528-47bd-b6a8-b42778e96d58","Type":"ContainerDied","Data":"e9437cf6b8ac103716ea5ea87fe94fb2a973c6966897518949dacf466e3c99f8"}
Nov 03 20:07:47.824900 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:47.824870    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f\": container with ID starting with 3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f not found: ID does not exist" containerID="3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f"
Nov 03 20:07:47.824988 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.824910    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f"} err="failed to get container status \"3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f\": rpc error: code = NotFound desc = could not find container \"3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f\": container with ID starting with 3749bab3848d79e35d642aaf00d503fd3e07f1eefeb7cc6537ce6d83dd70881f not found: ID does not exist"
Nov 03 20:07:47.824988 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.824931    4531 scope.go:122] "RemoveContainer" containerID="43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f"
Nov 03 20:07:47.839496 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.839469    4531 scope.go:122] "RemoveContainer" containerID="43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f"
Nov 03 20:07:47.839659 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.839636    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"]
Nov 03 20:07:47.839882 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:47.839861    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f\": container with ID starting with 43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f not found: ID does not exist" containerID="43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f"
Nov 03 20:07:47.839986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.839887    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f"} err="failed to get container status \"43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f\": rpc error: code = NotFound desc = could not find container \"43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f\": container with ID starting with 43df8d37f204f8141869f6213cacc43f6f4ba377c1efb56d06df91cfdb72881f not found: ID does not exist"
Nov 03 20:07:47.839986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.839907    4531 scope.go:122] "RemoveContainer" containerID="1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6"
Nov 03 20:07:47.844728 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.844692    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3ff278-76cbdc7bfc-hgc9h"]
Nov 03 20:07:47.868514 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.868492    4531 scope.go:122] "RemoveContainer" containerID="1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6"
Nov 03 20:07:47.869020 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:47.868900    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6\": container with ID starting with 1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6 not found: ID does not exist" containerID="1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6"
Nov 03 20:07:47.869020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.868935    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6"} err="failed to get container status \"1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6\": rpc error: code = NotFound desc = could not find container \"1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6\": container with ID starting with 1cba0b534bc8c2775756a8d3e5a316f5223f8547845a49afa28032b78124d6f6 not found: ID does not exist"
Nov 03 20:07:47.876356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.875865    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:47.876356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.875955    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"]
Nov 03 20:07:47.877233 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.877204    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c"]
Nov 03 20:07:47.883032 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.883002    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"]
Nov 03 20:07:47.890791 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:47.890764    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3ff278-79b8b9644c-jqjnd"]
Nov 03 20:07:48.170774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:48.170732    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-dce558/auth-test-model-dce558-kserve-74695d5dd5-xnv6c" podUID="8545f1c7-81a0-42a1-871e-47ca9a231c2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.241:8000/ready\": context deadline exceeded"
Nov 03 20:07:48.645021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:48.644982    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:48.858496 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:48.858463    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7c7784-0528-47bd-b6a8-b42778e96d58" path="/var/lib/kubelet/pods/0b7c7784-0528-47bd-b6a8-b42778e96d58/volumes"
Nov 03 20:07:48.858895 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:48.858882    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7987b86a-8744-4e09-a949-c23671975702" path="/var/lib/kubelet/pods/7987b86a-8744-4e09-a949-c23671975702/volumes"
Nov 03 20:07:48.859313 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:48.859299    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8545f1c7-81a0-42a1-871e-47ca9a231c2d" path="/var/lib/kubelet/pods/8545f1c7-81a0-42a1-871e-47ca9a231c2d/volumes"
Nov 03 20:07:49.344152 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:49.341009    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:49.989872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:49.989047    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:50.573388 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:50.573350    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:07:50.727815 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:50.727002    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:51.287613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.287576    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:51.415017 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.414973    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"]
Nov 03 20:07:51.415558 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415542    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a841ef7e-4a8d-469b-b13e-4d84980e3b1c" containerName="main"
Nov 03 20:07:51.415661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415558    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a841ef7e-4a8d-469b-b13e-4d84980e3b1c" containerName="main"
Nov 03 20:07:51.415661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415583    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8545f1c7-81a0-42a1-871e-47ca9a231c2d" containerName="main"
Nov 03 20:07:51.415661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415588    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8545f1c7-81a0-42a1-871e-47ca9a231c2d" containerName="main"
Nov 03 20:07:51.415661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415612    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0b7c7784-0528-47bd-b6a8-b42778e96d58" containerName="payload-pre-processing"
Nov 03 20:07:51.415661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415619    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7c7784-0528-47bd-b6a8-b42778e96d58" containerName="payload-pre-processing"
Nov 03 20:07:51.415661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415626    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7987b86a-8744-4e09-a949-c23671975702" containerName="payload-processing"
Nov 03 20:07:51.415661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415632    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7987b86a-8744-4e09-a949-c23671975702" containerName="payload-processing"
Nov 03 20:07:51.415661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415640    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8d1f628d-2a51-4bf7-a162-a800effecda7" containerName="revoke-keys"
Nov 03 20:07:51.415661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415645    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1f628d-2a51-4bf7-a162-a800effecda7" containerName="revoke-keys"
Nov 03 20:07:51.416010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415714    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7c7784-0528-47bd-b6a8-b42778e96d58" containerName="payload-pre-processing"
Nov 03 20:07:51.416010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415723    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7987b86a-8744-4e09-a949-c23671975702" containerName="payload-processing"
Nov 03 20:07:51.416010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415729    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1f628d-2a51-4bf7-a162-a800effecda7" containerName="revoke-keys"
Nov 03 20:07:51.416010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415737    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a841ef7e-4a8d-469b-b13e-4d84980e3b1c" containerName="main"
Nov 03 20:07:51.416010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.415744    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8545f1c7-81a0-42a1-871e-47ca9a231c2d" containerName="main"
Nov 03 20:07:51.421100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.420886    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:51.425708 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.425676    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"]
Nov 03 20:07:51.427312 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.427285    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:07:51.427472 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.427340    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:07:51.535965 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.535915    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:51.536218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.536079    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bqb9\" (UniqueName: \"kubernetes.io/projected/663e4167-e4b9-42d3-af01-10437f37e1e6-kube-api-access-2bqb9\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:51.536218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.536155    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:51.573128 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.573000    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:51.651225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.647012    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:51.651225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.647245    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:51.651225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.647350    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bqb9\" (UniqueName: \"kubernetes.io/projected/663e4167-e4b9-42d3-af01-10437f37e1e6-kube-api-access-2bqb9\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:51.651225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.649302    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:51.684193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.673023    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bqb9\" (UniqueName: \"kubernetes.io/projected/663e4167-e4b9-42d3-af01-10437f37e1e6-kube-api-access-2bqb9\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:51.684193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.673499    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:51.744268 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:51.744220    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:52.027226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:52.026653    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"]
Nov 03 20:07:52.030077 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:07:52.029594    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663e4167_e4b9_42d3_af01_10437f37e1e6.slice/crio-5924f9ad2934ece5fe94fe023c97c9875c757a266f217c767dc01a36e28dde18 WatchSource:0}: Error finding container 5924f9ad2934ece5fe94fe023c97c9875c757a266f217c767dc01a36e28dde18: Status 404 returned error can't find the container with id 5924f9ad2934ece5fe94fe023c97c9875c757a266f217c767dc01a36e28dde18
Nov 03 20:07:52.545757 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:52.545700    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:52.862498 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:52.862454    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6" event={"ID":"663e4167-e4b9-42d3-af01-10437f37e1e6","Type":"ContainerStarted","Data":"31ac6e22e768809912b6142281f055d9f8e488123bc188f77a9e60bb81bd3c5c"}
Nov 03 20:07:52.862894 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:52.862518    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6" event={"ID":"663e4167-e4b9-42d3-af01-10437f37e1e6","Type":"ContainerStarted","Data":"5924f9ad2934ece5fe94fe023c97c9875c757a266f217c767dc01a36e28dde18"}
Nov 03 20:07:52.877445 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:52.877387    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6" podStartSLOduration=1.877371558 podStartE2EDuration="1.877371558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:52.87529399 +0000 UTC m=+2186.145417297" watchObservedRunningTime="2026-11-03 20:07:52.877371558 +0000 UTC m=+2186.147494883"
Nov 03 20:07:53.327965 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:53.327927    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:53.474200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:53.473223    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:53.562767 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:53.562722    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"]
Nov 03 20:07:53.563027 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:53.562960    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" podUID="8d9e6662-96f2-42b0-853a-9649585967b5" containerName="istio-proxy" containerID="cri-o://48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65" gracePeriod=30
Nov 03 20:07:53.644687 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:53.644652    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:53.873568 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:53.873535    4531 generic.go:356] "Generic (PLEG): container finished" podID="663e4167-e4b9-42d3-af01-10437f37e1e6" containerID="31ac6e22e768809912b6142281f055d9f8e488123bc188f77a9e60bb81bd3c5c" exitCode=0
Nov 03 20:07:53.874120 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:53.873606    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6" event={"ID":"663e4167-e4b9-42d3-af01-10437f37e1e6","Type":"ContainerDied","Data":"31ac6e22e768809912b6142281f055d9f8e488123bc188f77a9e60bb81bd3c5c"}
Nov 03 20:07:54.328949 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:54.328890    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:55.033556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.033523    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:55.094557 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.094519    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663e4167-e4b9-42d3-af01-10437f37e1e6-kube-api-access-2bqb9\" (UniqueName: \"kubernetes.io/projected/663e4167-e4b9-42d3-af01-10437f37e1e6-kube-api-access-2bqb9\") pod \"663e4167-e4b9-42d3-af01-10437f37e1e6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") "
Nov 03 20:07:55.094557 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.094558    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-xks-ca\") pod \"663e4167-e4b9-42d3-af01-10437f37e1e6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") "
Nov 03 20:07:55.094778 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.094650    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-openshift-ca\") pod \"663e4167-e4b9-42d3-af01-10437f37e1e6\" (UID: \"663e4167-e4b9-42d3-af01-10437f37e1e6\") "
Nov 03 20:07:55.095025 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.095003    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-openshift-ca" pod "663e4167-e4b9-42d3-af01-10437f37e1e6" (UID: "663e4167-e4b9-42d3-af01-10437f37e1e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:55.096748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.096721    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-xks-ca" pod "663e4167-e4b9-42d3-af01-10437f37e1e6" (UID: "663e4167-e4b9-42d3-af01-10437f37e1e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:55.096748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.096731    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663e4167-e4b9-42d3-af01-10437f37e1e6-kube-api-access-2bqb9" pod "663e4167-e4b9-42d3-af01-10437f37e1e6" (UID: "663e4167-e4b9-42d3-af01-10437f37e1e6"). InnerVolumeSpecName "kube-api-access-2bqb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:55.155676 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.155296    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:55.196821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.196779    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:55.196821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.196816    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bqb9\" (UniqueName: \"kubernetes.io/projected/663e4167-e4b9-42d3-af01-10437f37e1e6-kube-api-access-2bqb9\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:55.196821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.196827    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663e4167-e4b9-42d3-af01-10437f37e1e6-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:55.892551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.892514    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6" event={"ID":"663e4167-e4b9-42d3-af01-10437f37e1e6","Type":"ContainerDied","Data":"5924f9ad2934ece5fe94fe023c97c9875c757a266f217c767dc01a36e28dde18"}
Nov 03 20:07:55.892751 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.892550    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"
Nov 03 20:07:55.892751 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.892571    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5924f9ad2934ece5fe94fe023c97c9875c757a266f217c767dc01a36e28dde18"
Nov 03 20:07:55.898978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:55.898942    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:56.542254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:56.542155    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:57.253843 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:57.250802    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:57.912870 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:57.912823    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:58.713315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.713275    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:58.833767 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.833705    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:07:58.914922 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.914889    4531 generic.go:356] "Generic (PLEG): container finished" podID="8d9e6662-96f2-42b0-853a-9649585967b5" containerID="48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65" exitCode=0
Nov 03 20:07:58.915406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.914938    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"
Nov 03 20:07:58.915406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.914972    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" event={"ID":"8d9e6662-96f2-42b0-853a-9649585967b5","Type":"ContainerDied","Data":"48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65"}
Nov 03 20:07:58.915406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.915032    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9" event={"ID":"8d9e6662-96f2-42b0-853a-9649585967b5","Type":"ContainerDied","Data":"7956992ffb46031e30d9e75e73a5f6b820a8e1130289e934a3965159d7a8f5f5"}
Nov 03 20:07:58.915406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.915063    4531 scope.go:122] "RemoveContainer" containerID="48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65"
Nov 03 20:07:58.925404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.925379    4531 scope.go:122] "RemoveContainer" containerID="48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65"
Nov 03 20:07:58.925686 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:07:58.925666    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65\": container with ID starting with 48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65 not found: ID does not exist" containerID="48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65"
Nov 03 20:07:58.925764 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.925693    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65"} err="failed to get container status \"48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65\": rpc error: code = NotFound desc = could not find container \"48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65\": container with ID starting with 48fca4284d2523b4b4b1c2ce80f9e17bb7895ae7d61a0609ced065f123cf6c65 not found: ID does not exist"
Nov 03 20:07:58.944206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944135    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-certs\") pod \"8d9e6662-96f2-42b0-853a-9649585967b5\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") "
Nov 03 20:07:58.944206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944255    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d9e6662-96f2-42b0-853a-9649585967b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9e6662-96f2-42b0-853a-9649585967b5-istiod-ca-cert\") pod \"8d9e6662-96f2-42b0-853a-9649585967b5\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") "
Nov 03 20:07:58.944521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944324    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-istio-token\" (UniqueName: \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-istio-token\") pod \"8d9e6662-96f2-42b0-853a-9649585967b5\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") "
Nov 03 20:07:58.944521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944393    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-envoy\") pod \"8d9e6662-96f2-42b0-853a-9649585967b5\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") "
Nov 03 20:07:58.944521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944469    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-kube-api-access-rq2pp\" (UniqueName: \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-kube-api-access-rq2pp\") pod \"8d9e6662-96f2-42b0-853a-9649585967b5\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") "
Nov 03 20:07:58.944521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944510    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-socket\") pod \"8d9e6662-96f2-42b0-853a-9649585967b5\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") "
Nov 03 20:07:58.944746 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944549    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-data\") pod \"8d9e6662-96f2-42b0-853a-9649585967b5\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") "
Nov 03 20:07:58.944746 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944587    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d9e6662-96f2-42b0-853a-9649585967b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9e6662-96f2-42b0-853a-9649585967b5-istio-podinfo\") pod \"8d9e6662-96f2-42b0-853a-9649585967b5\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") "
Nov 03 20:07:58.944746 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944548    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-certs" pod "8d9e6662-96f2-42b0-853a-9649585967b5" (UID: "8d9e6662-96f2-42b0-853a-9649585967b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:58.944746 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944649    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-credential-socket\") pod \"8d9e6662-96f2-42b0-853a-9649585967b5\" (UID: \"8d9e6662-96f2-42b0-853a-9649585967b5\") "
Nov 03 20:07:58.945008 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944844    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-data" pod "8d9e6662-96f2-42b0-853a-9649585967b5" (UID: "8d9e6662-96f2-42b0-853a-9649585967b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:58.945008 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.944860    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9e6662-96f2-42b0-853a-9649585967b5-istiod-ca-cert" pod "8d9e6662-96f2-42b0-853a-9649585967b5" (UID: "8d9e6662-96f2-42b0-853a-9649585967b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:58.945120 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.945063    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-credential-socket" pod "8d9e6662-96f2-42b0-853a-9649585967b5" (UID: "8d9e6662-96f2-42b0-853a-9649585967b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:58.945120 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.945076    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-socket" pod "8d9e6662-96f2-42b0-853a-9649585967b5" (UID: "8d9e6662-96f2-42b0-853a-9649585967b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:58.945484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.945370    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9e6662-96f2-42b0-853a-9649585967b5-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:58.945484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.945396    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:58.945484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.945411    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:58.945484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.945425    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:58.945484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.945440    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:58.946884 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.946861    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-istio-token" pod "8d9e6662-96f2-42b0-853a-9649585967b5" (UID: "8d9e6662-96f2-42b0-853a-9649585967b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:58.946982 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.946949    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-kube-api-access-rq2pp" pod "8d9e6662-96f2-42b0-853a-9649585967b5" (UID: "8d9e6662-96f2-42b0-853a-9649585967b5"). InnerVolumeSpecName "kube-api-access-rq2pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:58.947057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.947041    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-envoy" pod "8d9e6662-96f2-42b0-853a-9649585967b5" (UID: "8d9e6662-96f2-42b0-853a-9649585967b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:58.947187 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:58.947158    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d9e6662-96f2-42b0-853a-9649585967b5-istio-podinfo" pod "8d9e6662-96f2-42b0-853a-9649585967b5" (UID: "8d9e6662-96f2-42b0-853a-9649585967b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:07:59.046781 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:59.046745    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq2pp\" (UniqueName: \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-kube-api-access-rq2pp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:59.046781 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:59.046778    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9e6662-96f2-42b0-853a-9649585967b5-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:59.046781 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:59.046788    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9e6662-96f2-42b0-853a-9649585967b5-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:59.047048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:59.046796    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9e6662-96f2-42b0-853a-9649585967b5-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:59.238973 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:59.238931    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"]
Nov 03 20:07:59.245694 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:59.244804    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-fwlw9"]
Nov 03 20:07:59.411952 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:07:59.411902    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:00.036325 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:00.035763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:00.738202 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:00.734626    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:00.857561 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:00.857530    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d9e6662-96f2-42b0-853a-9649585967b5" path="/var/lib/kubelet/pods/8d9e6662-96f2-42b0-853a-9649585967b5/volumes"
Nov 03 20:08:01.427131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:01.427075    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:01.682554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:01.682469    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:01.810032 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:01.809991    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:01.959073 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:01.958991    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:02.110403 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:02.110350    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:02.208827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:02.208788    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"]
Nov 03 20:08:02.211825 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:02.211753    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-801162-85ng6"]
Nov 03 20:08:02.278227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:02.278162    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:02.413441 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:02.413403    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:02.729002 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:02.728968    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:02.858468 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:02.858435    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663e4167-e4b9-42d3-af01-10437f37e1e6" path="/var/lib/kubelet/pods/663e4167-e4b9-42d3-af01-10437f37e1e6/volumes"
Nov 03 20:08:03.238056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:03.238016    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:03.988166 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:03.988125    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:04.424337 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:04.424293    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:04.554254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:04.554219    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:04.682358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:04.682261    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:04.863257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:04.861190    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:05.510582 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:05.510538    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:06.152308 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:06.152261    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:06.830012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:06.829970    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:07.441125 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:07.441089    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:07.584916 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:07.584882    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:07.743298 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:07.743200    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:08.038878 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:08.038794    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:08.435059 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:08.435028    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.311130 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:09.311093    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.549413 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:09.549317    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.689705 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:09.689673    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.834903 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:09.834867    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:10.150310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:10.150262    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:11.104076 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:11.101575    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:11.744424 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:11.743522    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:12.420279 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:12.420244    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:13.240694 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:13.240656    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:13.851145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:13.851108    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:14.564202 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:14.562091    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:14.697893 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:14.697861    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:14.925219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:14.925153    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"]
Nov 03 20:08:14.925689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:14.925468    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy" containerID="cri-o://124c9effe4b79c2d4ca27b95b757a7c046ddfbd4b407cd5a321a8de1a2e67a77" gracePeriod=30
Nov 03 20:08:15.010209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:15.009241    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:15.318772 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:15.318683    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:15.465121 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:15.465084    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:16.167991 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:16.167947    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:08:16.168425 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:16.168021    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:08:16.461926 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:16.461833    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:17.206156 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:17.206121    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:17.407917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:17.407846    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:18.165244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:18.157456    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:18.945854 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:18.945758    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:19.719759 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:19.717741    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:20.093236 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.093200    4531 generic.go:356] "Generic (PLEG): container finished" podID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerID="124c9effe4b79c2d4ca27b95b757a7c046ddfbd4b407cd5a321a8de1a2e67a77" exitCode=0
Nov 03 20:08:20.093650 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.093264    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" event={"ID":"c9bcb497-8b2e-4fcd-9e14-110914fe6cde","Type":"ContainerDied","Data":"124c9effe4b79c2d4ca27b95b757a7c046ddfbd4b407cd5a321a8de1a2e67a77"}
Nov 03 20:08:20.196124 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.196099    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:08:20.302469 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302382    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-credential-socket\") pod \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") "
Nov 03 20:08:20.302469 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302449    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-socket\") pod \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") "
Nov 03 20:08:20.302730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302507    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-token\" (UniqueName: \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-token\") pod \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") "
Nov 03 20:08:20.302730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302539    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istiod-ca-cert\") pod \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") "
Nov 03 20:08:20.302730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302578    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-data\") pod \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") "
Nov 03 20:08:20.302730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302602    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-envoy\") pod \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") "
Nov 03 20:08:20.302730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302631    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-certs\") pod \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") "
Nov 03 20:08:20.302730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302654    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-kube-api-access-psxf7\" (UniqueName: \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-kube-api-access-psxf7\") pod \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") "
Nov 03 20:08:20.302730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302695    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-podinfo\") pod \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\" (UID: \"c9bcb497-8b2e-4fcd-9e14-110914fe6cde\") "
Nov 03 20:08:20.302730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302713    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-credential-socket" pod "c9bcb497-8b2e-4fcd-9e14-110914fe6cde" (UID: "c9bcb497-8b2e-4fcd-9e14-110914fe6cde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:20.303133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302825    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-socket" pod "c9bcb497-8b2e-4fcd-9e14-110914fe6cde" (UID: "c9bcb497-8b2e-4fcd-9e14-110914fe6cde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:20.303133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.302945    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-data" pod "c9bcb497-8b2e-4fcd-9e14-110914fe6cde" (UID: "c9bcb497-8b2e-4fcd-9e14-110914fe6cde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:20.303133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.303039    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istiod-ca-cert" pod "c9bcb497-8b2e-4fcd-9e14-110914fe6cde" (UID: "c9bcb497-8b2e-4fcd-9e14-110914fe6cde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:20.303358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.303235    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.303358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.303341    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-certs" pod "c9bcb497-8b2e-4fcd-9e14-110914fe6cde" (UID: "c9bcb497-8b2e-4fcd-9e14-110914fe6cde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:20.303478 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.303359    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.303478 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.303377    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.303478 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.303390    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.305017 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.304982    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-envoy" pod "c9bcb497-8b2e-4fcd-9e14-110914fe6cde" (UID: "c9bcb497-8b2e-4fcd-9e14-110914fe6cde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:20.305017 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.304989    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-token" pod "c9bcb497-8b2e-4fcd-9e14-110914fe6cde" (UID: "c9bcb497-8b2e-4fcd-9e14-110914fe6cde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:20.305246 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.305098    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-kube-api-access-psxf7" pod "c9bcb497-8b2e-4fcd-9e14-110914fe6cde" (UID: "c9bcb497-8b2e-4fcd-9e14-110914fe6cde"). InnerVolumeSpecName "kube-api-access-psxf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:20.305246 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.305133    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-podinfo" pod "c9bcb497-8b2e-4fcd-9e14-110914fe6cde" (UID: "c9bcb497-8b2e-4fcd-9e14-110914fe6cde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:08:20.350073 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.350023    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"]
Nov 03 20:08:20.370715 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.370677    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"]
Nov 03 20:08:20.405494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.405454    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.405494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.405496    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.405756 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.405513    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psxf7\" (UniqueName: \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-kube-api-access-psxf7\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.405756 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.405531    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.405756 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.405716    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bcb497-8b2e-4fcd-9e14-110914fe6cde-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.461245 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.461207    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:20.574286 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:20.574165    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:08:20.609919 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:20.609882    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:21.066737 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:21.066699    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:21.101094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:21.101062    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"
Nov 03 20:08:21.101543 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:21.101061    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw" event={"ID":"c9bcb497-8b2e-4fcd-9e14-110914fe6cde","Type":"ContainerDied","Data":"b815691762cfcd7f70b3a5b4ec2495944b77c10658fbc251173bf9fd4f9773a7"}
Nov 03 20:08:21.101543 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:21.101218    4531 scope.go:122] "RemoveContainer" containerID="124c9effe4b79c2d4ca27b95b757a7c046ddfbd4b407cd5a321a8de1a2e67a77"
Nov 03 20:08:21.120202 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:21.119987    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"]
Nov 03 20:08:21.123068 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:21.123038    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-fvwmw"]
Nov 03 20:08:21.213631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:21.213595    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:21.334519 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:21.334486    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:21.499204 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:21.498867    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:21.827672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:21.827558    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:22.020002 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:22.019959    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"]
Nov 03 20:08:22.022882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:22.022850    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55674fc2-jj76l"]
Nov 03 20:08:22.253025 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:22.252014    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:22.857567 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:22.857523    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ecbfff-be37-40a2-90cb-bf95ab9c5a28" path="/var/lib/kubelet/pods/38ecbfff-be37-40a2-90cb-bf95ab9c5a28/volumes"
Nov 03 20:08:22.858136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:22.858114    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" path="/var/lib/kubelet/pods/c9bcb497-8b2e-4fcd-9e14-110914fe6cde/volumes"
Nov 03 20:08:22.950837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:22.950798    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:23.644207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:23.643463    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:24.422931 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:24.421264    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:25.245474 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:25.245432    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:25.677056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:25.677016    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"]
Nov 03 20:08:25.801579 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:25.801526    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:25.929468 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:25.929377    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:26.060777 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.060735    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:26.469880 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.469843    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"]
Nov 03 20:08:26.470454 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.470431    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8d9e6662-96f2-42b0-853a-9649585967b5" containerName="istio-proxy"
Nov 03 20:08:26.470531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.470458    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9e6662-96f2-42b0-853a-9649585967b5" containerName="istio-proxy"
Nov 03 20:08:26.470531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.470483    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy"
Nov 03 20:08:26.470531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.470488    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy"
Nov 03 20:08:26.470531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.470514    4531 cpu_manager.go:414] "containerMap: removing container" podUID="663e4167-e4b9-42d3-af01-10437f37e1e6" containerName="revoke-keys"
Nov 03 20:08:26.470531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.470520    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663e4167-e4b9-42d3-af01-10437f37e1e6" containerName="revoke-keys"
Nov 03 20:08:26.470749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.470598    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9e6662-96f2-42b0-853a-9649585967b5" containerName="istio-proxy"
Nov 03 20:08:26.470749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.470608    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bcb497-8b2e-4fcd-9e14-110914fe6cde" containerName="istio-proxy"
Nov 03 20:08:26.470749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.470618    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="663e4167-e4b9-42d3-af01-10437f37e1e6" containerName="revoke-keys"
Nov 03 20:08:26.474333 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.474305    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:26.477925 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.477894    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:08:26.478861 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.478620    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:08:26.481654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.481626    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"]
Nov 03 20:08:26.585262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.585213    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hkk\" (UniqueName: \"kubernetes.io/projected/9683b009-84b8-4509-b43b-1a18d6646f7c-kube-api-access-z4hkk\") pod \"maas-api-revoke-sub-2215c816-ptcqm\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:26.585471 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.585415    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2215c816-ptcqm\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:26.585571 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.585552    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2215c816-ptcqm\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:26.686670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.686628    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2215c816-ptcqm\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:26.686882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.686739    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2215c816-ptcqm\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:26.686882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.686777    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hkk\" (UniqueName: \"kubernetes.io/projected/9683b009-84b8-4509-b43b-1a18d6646f7c-kube-api-access-z4hkk\") pod \"maas-api-revoke-sub-2215c816-ptcqm\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:26.687382 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.687353    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2215c816-ptcqm\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:26.689352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.689320    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2215c816-ptcqm\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:26.693856 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.693833    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hkk\" (UniqueName: \"kubernetes.io/projected/9683b009-84b8-4509-b43b-1a18d6646f7c-kube-api-access-z4hkk\") pod \"maas-api-revoke-sub-2215c816-ptcqm\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:26.798009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.797922    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:26.840760 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:26.840714    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:27.102554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:27.102521    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"]
Nov 03 20:08:27.103798 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:08:27.103775    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9683b009_84b8_4509_b43b_1a18d6646f7c.slice/crio-d9ff0a21074626fed9b8b3ff4c12d8042c662629a6ad542cc11ed612ca6f6701 WatchSource:0}: Error finding container d9ff0a21074626fed9b8b3ff4c12d8042c662629a6ad542cc11ed612ca6f6701: Status 404 returned error can't find the container with id d9ff0a21074626fed9b8b3ff4c12d8042c662629a6ad542cc11ed612ca6f6701
Nov 03 20:08:27.145030 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:27.144997    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm" event={"ID":"9683b009-84b8-4509-b43b-1a18d6646f7c","Type":"ContainerStarted","Data":"d9ff0a21074626fed9b8b3ff4c12d8042c662629a6ad542cc11ed612ca6f6701"}
Nov 03 20:08:27.522531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:27.522484    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:27.886531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:27.886499    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:28.154996 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:28.154912    4531 generic.go:356] "Generic (PLEG): container finished" podID="9683b009-84b8-4509-b43b-1a18d6646f7c" containerID="5a427733517b60daff8a37d6eb5438f0606cf74853cde9e4d380ad30de2c89c5" exitCode=0
Nov 03 20:08:28.155162 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:28.154994    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm" event={"ID":"9683b009-84b8-4509-b43b-1a18d6646f7c","Type":"ContainerDied","Data":"5a427733517b60daff8a37d6eb5438f0606cf74853cde9e4d380ad30de2c89c5"}
Nov 03 20:08:28.256220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:28.253300    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:28.383507 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:28.383473    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:28.486451 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:28.486353    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"]
Nov 03 20:08:28.486644 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:28.486555    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy" containerID="cri-o://7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19" gracePeriod=30
Nov 03 20:08:28.565501 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:28.565455    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:28.928443 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:28.928396    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:29.076111 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.076076    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:29.303216 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.303160    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:29.422818 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.422782    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-openshift-ca\") pod \"9683b009-84b8-4509-b43b-1a18d6646f7c\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") "
Nov 03 20:08:29.423016 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.422841    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9683b009-84b8-4509-b43b-1a18d6646f7c-kube-api-access-z4hkk\" (UniqueName: \"kubernetes.io/projected/9683b009-84b8-4509-b43b-1a18d6646f7c-kube-api-access-z4hkk\") pod \"9683b009-84b8-4509-b43b-1a18d6646f7c\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") "
Nov 03 20:08:29.423064 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.423032    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-xks-ca\") pod \"9683b009-84b8-4509-b43b-1a18d6646f7c\" (UID: \"9683b009-84b8-4509-b43b-1a18d6646f7c\") "
Nov 03 20:08:29.423267 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.423240    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-openshift-ca" pod "9683b009-84b8-4509-b43b-1a18d6646f7c" (UID: "9683b009-84b8-4509-b43b-1a18d6646f7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:29.423503 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.423475    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:29.425206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.425165    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9683b009-84b8-4509-b43b-1a18d6646f7c-kube-api-access-z4hkk" pod "9683b009-84b8-4509-b43b-1a18d6646f7c" (UID: "9683b009-84b8-4509-b43b-1a18d6646f7c"). InnerVolumeSpecName "kube-api-access-z4hkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:29.425206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.425164    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-xks-ca" pod "9683b009-84b8-4509-b43b-1a18d6646f7c" (UID: "9683b009-84b8-4509-b43b-1a18d6646f7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:29.525109 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.525013    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4hkk\" (UniqueName: \"kubernetes.io/projected/9683b009-84b8-4509-b43b-1a18d6646f7c-kube-api-access-z4hkk\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:29.525109 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.525051    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9683b009-84b8-4509-b43b-1a18d6646f7c-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:29.904042 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:29.904005    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:30.170799 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:30.170707    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"
Nov 03 20:08:30.170799 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:30.170742    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm" event={"ID":"9683b009-84b8-4509-b43b-1a18d6646f7c","Type":"ContainerDied","Data":"d9ff0a21074626fed9b8b3ff4c12d8042c662629a6ad542cc11ed612ca6f6701"}
Nov 03 20:08:30.171008 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:30.170800    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9ff0a21074626fed9b8b3ff4c12d8042c662629a6ad542cc11ed612ca6f6701"
Nov 03 20:08:30.677522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:30.677477    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5" podUID="0507af87-0ab8-46b3-9c18-45a7af463894" containerName="payload-pre-processing" containerID="cri-o://c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03" gracePeriod=25
Nov 03 20:08:30.796445 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:30.796404    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:30.955275 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:30.955239    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 20:08:31.042652 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.042609    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0507af87-0ab8-46b3-9c18-45a7af463894-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0507af87-0ab8-46b3-9c18-45a7af463894-plugins-config-volume\") pod \"0507af87-0ab8-46b3-9c18-45a7af463894\" (UID: \"0507af87-0ab8-46b3-9c18-45a7af463894\") "
Nov 03 20:08:31.042849 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.042682    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0507af87-0ab8-46b3-9c18-45a7af463894-kube-api-access-6b4nx\" (UniqueName: \"kubernetes.io/projected/0507af87-0ab8-46b3-9c18-45a7af463894-kube-api-access-6b4nx\") pod \"0507af87-0ab8-46b3-9c18-45a7af463894\" (UID: \"0507af87-0ab8-46b3-9c18-45a7af463894\") "
Nov 03 20:08:31.043110 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.043084    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0507af87-0ab8-46b3-9c18-45a7af463894-plugins-config-volume" pod "0507af87-0ab8-46b3-9c18-45a7af463894" (UID: "0507af87-0ab8-46b3-9c18-45a7af463894"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:31.044890 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.044863    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0507af87-0ab8-46b3-9c18-45a7af463894-kube-api-access-6b4nx" pod "0507af87-0ab8-46b3-9c18-45a7af463894" (UID: "0507af87-0ab8-46b3-9c18-45a7af463894"). InnerVolumeSpecName "kube-api-access-6b4nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:31.143946 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.143899    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0507af87-0ab8-46b3-9c18-45a7af463894-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:31.143946 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.143939    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b4nx\" (UniqueName: \"kubernetes.io/projected/0507af87-0ab8-46b3-9c18-45a7af463894-kube-api-access-6b4nx\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:31.179683 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.179643    4531 generic.go:356] "Generic (PLEG): container finished" podID="0507af87-0ab8-46b3-9c18-45a7af463894" containerID="c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03" exitCode=0
Nov 03 20:08:31.179844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.179696    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"
Nov 03 20:08:31.179844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.179730    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5" event={"ID":"0507af87-0ab8-46b3-9c18-45a7af463894","Type":"ContainerDied","Data":"c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03"}
Nov 03 20:08:31.179844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.179791    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5" event={"ID":"0507af87-0ab8-46b3-9c18-45a7af463894","Type":"ContainerDied","Data":"cdd14945467d2b34e53ae2ad8a991255140782b6a63465c3575cefc26344b9a0"}
Nov 03 20:08:31.179844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.179813    4531 scope.go:122] "RemoveContainer" containerID="c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03"
Nov 03 20:08:31.191933 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.191911    4531 scope.go:122] "RemoveContainer" containerID="c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03"
Nov 03 20:08:31.192296 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:31.192270    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03\": container with ID starting with c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03 not found: ID does not exist" containerID="c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03"
Nov 03 20:08:31.192366 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.192306    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03"} err="failed to get container status \"c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03\": rpc error: code = NotFound desc = could not find container \"c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03\": container with ID starting with c9d9d6695a7a643a63b4142283090d27187d87c104b57720f1c03d2b312a3d03 not found: ID does not exist"
Nov 03 20:08:31.204875 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.204842    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"]
Nov 03 20:08:31.209902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.209872    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-82dbbf-76cc9bd4f9-zlkq5"]
Nov 03 20:08:31.635300 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:31.628280    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:32.429650 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:32.428950    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:32.857917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:32.857835    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0507af87-0ab8-46b3-9c18-45a7af463894" path="/var/lib/kubelet/pods/0507af87-0ab8-46b3-9c18-45a7af463894/volumes"
Nov 03 20:08:33.122771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.122653    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:33.755911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.755850    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:08:33.840745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.840704    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:33.878522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.878485    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istiod-ca-cert\") pod \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") "
Nov 03 20:08:33.878710 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.878600    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-credential-socket\") pod \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") "
Nov 03 20:08:33.878710 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.878642    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-token\" (UniqueName: \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-token\") pod \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") "
Nov 03 20:08:33.878710 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.878677    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-certs\") pod \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") "
Nov 03 20:08:33.878972 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.878713    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-podinfo\") pod \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") "
Nov 03 20:08:33.878972 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.878754    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-socket\") pod \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") "
Nov 03 20:08:33.878972 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.878783    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-kube-api-access-9fkkv\" (UniqueName: \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-kube-api-access-9fkkv\") pod \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") "
Nov 03 20:08:33.878972 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.878853    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-data\") pod \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") "
Nov 03 20:08:33.878972 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.878945    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-envoy\") pod \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\" (UID: \"7e747c83-6f8d-47ac-8023-d86ce46bcf94\") "
Nov 03 20:08:33.878972 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.878954    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-credential-socket" pod "7e747c83-6f8d-47ac-8023-d86ce46bcf94" (UID: "7e747c83-6f8d-47ac-8023-d86ce46bcf94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:33.879362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.878993    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istiod-ca-cert" pod "7e747c83-6f8d-47ac-8023-d86ce46bcf94" (UID: "7e747c83-6f8d-47ac-8023-d86ce46bcf94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:33.879362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.879132    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-socket" pod "7e747c83-6f8d-47ac-8023-d86ce46bcf94" (UID: "7e747c83-6f8d-47ac-8023-d86ce46bcf94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:33.879362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.879275    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-data" pod "7e747c83-6f8d-47ac-8023-d86ce46bcf94" (UID: "7e747c83-6f8d-47ac-8023-d86ce46bcf94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:33.879491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.879448    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:33.879491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.879470    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:33.879491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.879484    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:33.879657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.879497    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:33.879867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.879839    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-certs" pod "7e747c83-6f8d-47ac-8023-d86ce46bcf94" (UID: "7e747c83-6f8d-47ac-8023-d86ce46bcf94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:33.881486 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.881442    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-token" pod "7e747c83-6f8d-47ac-8023-d86ce46bcf94" (UID: "7e747c83-6f8d-47ac-8023-d86ce46bcf94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:33.881588 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.881533    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-envoy" pod "7e747c83-6f8d-47ac-8023-d86ce46bcf94" (UID: "7e747c83-6f8d-47ac-8023-d86ce46bcf94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:33.882145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.882082    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-kube-api-access-9fkkv" pod "7e747c83-6f8d-47ac-8023-d86ce46bcf94" (UID: "7e747c83-6f8d-47ac-8023-d86ce46bcf94"). InnerVolumeSpecName "kube-api-access-9fkkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:33.882426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.882393    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-podinfo" pod "7e747c83-6f8d-47ac-8023-d86ce46bcf94" (UID: "7e747c83-6f8d-47ac-8023-d86ce46bcf94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:08:33.927107 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.927058    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"]
Nov 03 20:08:33.933077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.933047    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"]
Nov 03 20:08:33.949907 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.949869    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"]
Nov 03 20:08:33.956548 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.956517    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"]
Nov 03 20:08:33.980602 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.980572    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:33.980602 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.980599    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fkkv\" (UniqueName: \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-kube-api-access-9fkkv\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:33.980602 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.980611    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:33.980602 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.980620    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e747c83-6f8d-47ac-8023-d86ce46bcf94-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:33.981077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:33.980628    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e747c83-6f8d-47ac-8023-d86ce46bcf94-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:34.080383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.080299    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:34.214280 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.214246    4531 generic.go:356] "Generic (PLEG): container finished" podID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerID="7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19" exitCode=0
Nov 03 20:08:34.214488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.214325    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"
Nov 03 20:08:34.214488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.214337    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" event={"ID":"7e747c83-6f8d-47ac-8023-d86ce46bcf94","Type":"ContainerDied","Data":"7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19"}
Nov 03 20:08:34.214488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.214400    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc" event={"ID":"7e747c83-6f8d-47ac-8023-d86ce46bcf94","Type":"ContainerDied","Data":"778af0998d328be0b08a735be97bbfd66eb824bdcb3268eafb4d42ee885b1b2e"}
Nov 03 20:08:34.214488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.214430    4531 scope.go:122] "RemoveContainer" containerID="7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19"
Nov 03 20:08:34.226971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.226951    4531 scope.go:122] "RemoveContainer" containerID="58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982"
Nov 03 20:08:34.238088 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.238068    4531 scope.go:122] "RemoveContainer" containerID="7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19"
Nov 03 20:08:34.238441 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:34.238417    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19\": container with ID starting with 7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19 not found: ID does not exist" containerID="7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19"
Nov 03 20:08:34.238558 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.238448    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19"} err="failed to get container status \"7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19\": rpc error: code = NotFound desc = could not find container \"7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19\": container with ID starting with 7e81eb18c11eb166fd81ed952a3caea9585af8e2662b2ad8acdf3c131de9cf19 not found: ID does not exist"
Nov 03 20:08:34.238558 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.238468    4531 scope.go:122] "RemoveContainer" containerID="58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982"
Nov 03 20:08:34.238753 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:34.238731    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982\": container with ID starting with 58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982 not found: ID does not exist" containerID="58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982"
Nov 03 20:08:34.238794 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.238759    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982"} err="failed to get container status \"58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982\": rpc error: code = NotFound desc = could not find container \"58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982\": container with ID starting with 58cf7c5885756d8c032fa20dc4d243fe721d77e71044f7563586a8f0888ca982 not found: ID does not exist"
Nov 03 20:08:34.249064 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.249035    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"]
Nov 03 20:08:34.253122 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.253090    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-ln4gc"]
Nov 03 20:08:34.635474 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.635426    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:34.858926 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:34.858886    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" path="/var/lib/kubelet/pods/7e747c83-6f8d-47ac-8023-d86ce46bcf94/volumes"
Nov 03 20:08:35.255662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.255621    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:35.371975 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.371919    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main" containerID="cri-o://47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528" gracePeriod=45
Nov 03 20:08:35.393760 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.393725    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main" containerID="cri-o://0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1" gracePeriod=45
Nov 03 20:08:35.601452 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.601408    4531 scope.go:122] "RemoveContainer" containerID="aea1288817bdf2db7bdd12c4e26782d2aee244139e82e0518c6da7cdb55eaa03"
Nov 03 20:08:35.623805 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.621680    4531 scope.go:122] "RemoveContainer" containerID="75d417d35ba14c58c40faf4172c7786ad40020f806213662932f3b92cb9244fa"
Nov 03 20:08:35.645814 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.645790    4531 scope.go:122] "RemoveContainer" containerID="1e8c3ed107673db6d3fe42cc6c5cf800bb2f361e2c56b5a03fc339be76c23308"
Nov 03 20:08:35.661464 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.661437    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 20:08:35.699670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.699640    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 20:08:35.800958 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.800872    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8306ce04-6749-4236-9359-895705d5fbe1-kube-api-access-vbdck\" (UniqueName: \"kubernetes.io/projected/8306ce04-6749-4236-9359-895705d5fbe1-kube-api-access-vbdck\") pod \"8306ce04-6749-4236-9359-895705d5fbe1\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") "
Nov 03 20:08:35.800958 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.800930    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44825f51-d618-4095-93fe-478a2f497eeb-kube-api-access-fnpjm\" (UniqueName: \"kubernetes.io/projected/44825f51-d618-4095-93fe-478a2f497eeb-kube-api-access-fnpjm\") pod \"44825f51-d618-4095-93fe-478a2f497eeb\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") "
Nov 03 20:08:35.801167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.800965    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-home\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-home\") pod \"8306ce04-6749-4236-9359-895705d5fbe1\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") "
Nov 03 20:08:35.801167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801144    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44825f51-d618-4095-93fe-478a2f497eeb-tls-certs\" (UniqueName: \"kubernetes.io/secret/44825f51-d618-4095-93fe-478a2f497eeb-tls-certs\") pod \"44825f51-d618-4095-93fe-478a2f497eeb\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") "
Nov 03 20:08:35.801323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801227    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-home\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-home\") pod \"44825f51-d618-4095-93fe-478a2f497eeb\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") "
Nov 03 20:08:35.801323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801274    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-dshm\") pod \"44825f51-d618-4095-93fe-478a2f497eeb\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") "
Nov 03 20:08:35.801438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801227    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-home" pod "8306ce04-6749-4236-9359-895705d5fbe1" (UID: "8306ce04-6749-4236-9359-895705d5fbe1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.801438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801335    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-tmp-dir\") pod \"44825f51-d618-4095-93fe-478a2f497eeb\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") "
Nov 03 20:08:35.801438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801390    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-model-cache\") pod \"8306ce04-6749-4236-9359-895705d5fbe1\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") "
Nov 03 20:08:35.801615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801438    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-home" pod "44825f51-d618-4095-93fe-478a2f497eeb" (UID: "44825f51-d618-4095-93fe-478a2f497eeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.801615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801441    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8306ce04-6749-4236-9359-895705d5fbe1-tls-certs\" (UniqueName: \"kubernetes.io/secret/8306ce04-6749-4236-9359-895705d5fbe1-tls-certs\") pod \"8306ce04-6749-4236-9359-895705d5fbe1\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") "
Nov 03 20:08:35.801615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801528    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-dshm\") pod \"8306ce04-6749-4236-9359-895705d5fbe1\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") "
Nov 03 20:08:35.801804 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801682    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-tmp-dir\") pod \"8306ce04-6749-4236-9359-895705d5fbe1\" (UID: \"8306ce04-6749-4236-9359-895705d5fbe1\") "
Nov 03 20:08:35.801804 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801736    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-model-cache\") pod \"44825f51-d618-4095-93fe-478a2f497eeb\" (UID: \"44825f51-d618-4095-93fe-478a2f497eeb\") "
Nov 03 20:08:35.801943 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.801907    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-model-cache" pod "8306ce04-6749-4236-9359-895705d5fbe1" (UID: "8306ce04-6749-4236-9359-895705d5fbe1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.802437 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.802211    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-tmp-dir" pod "8306ce04-6749-4236-9359-895705d5fbe1" (UID: "8306ce04-6749-4236-9359-895705d5fbe1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.802437 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.802384    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-tmp-dir" pod "44825f51-d618-4095-93fe-478a2f497eeb" (UID: "44825f51-d618-4095-93fe-478a2f497eeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.802437 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.802402    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-model-cache" pod "44825f51-d618-4095-93fe-478a2f497eeb" (UID: "44825f51-d618-4095-93fe-478a2f497eeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.802702 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.802447    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.802702 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.802471    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.802702 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.802486    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.802702 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.802500    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.803957 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.803932    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8306ce04-6749-4236-9359-895705d5fbe1-tls-certs" pod "8306ce04-6749-4236-9359-895705d5fbe1" (UID: "8306ce04-6749-4236-9359-895705d5fbe1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:35.804147 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.804044    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44825f51-d618-4095-93fe-478a2f497eeb-tls-certs" pod "44825f51-d618-4095-93fe-478a2f497eeb" (UID: "44825f51-d618-4095-93fe-478a2f497eeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:35.804235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.804146    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8306ce04-6749-4236-9359-895705d5fbe1-kube-api-access-vbdck" pod "8306ce04-6749-4236-9359-895705d5fbe1" (UID: "8306ce04-6749-4236-9359-895705d5fbe1"). InnerVolumeSpecName "kube-api-access-vbdck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:35.804235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.804154    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-dshm" pod "44825f51-d618-4095-93fe-478a2f497eeb" (UID: "44825f51-d618-4095-93fe-478a2f497eeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.804415 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.804398    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44825f51-d618-4095-93fe-478a2f497eeb-kube-api-access-fnpjm" pod "44825f51-d618-4095-93fe-478a2f497eeb" (UID: "44825f51-d618-4095-93fe-478a2f497eeb"). InnerVolumeSpecName "kube-api-access-fnpjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:35.804452 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.804425    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-dshm" pod "8306ce04-6749-4236-9359-895705d5fbe1" (UID: "8306ce04-6749-4236-9359-895705d5fbe1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.903797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.903749    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.903797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.903789    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbdck\" (UniqueName: \"kubernetes.io/projected/8306ce04-6749-4236-9359-895705d5fbe1-kube-api-access-vbdck\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.903797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.903802    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnpjm\" (UniqueName: \"kubernetes.io/projected/44825f51-d618-4095-93fe-478a2f497eeb-kube-api-access-fnpjm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.904340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.903812    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44825f51-d618-4095-93fe-478a2f497eeb-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.904340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.903822    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.904340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.903833    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44825f51-d618-4095-93fe-478a2f497eeb-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.904340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.903842    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8306ce04-6749-4236-9359-895705d5fbe1-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.904340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:35.903849    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8306ce04-6749-4236-9359-895705d5fbe1-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:36.091303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.079644    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:36.232773 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.232737    4531 generic.go:356] "Generic (PLEG): container finished" podID="44825f51-d618-4095-93fe-478a2f497eeb" containerID="0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1" exitCode=0
Nov 03 20:08:36.232989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.232781    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"
Nov 03 20:08:36.232989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.232817    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" event={"ID":"44825f51-d618-4095-93fe-478a2f497eeb","Type":"ContainerDied","Data":"0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1"}
Nov 03 20:08:36.232989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.232858    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw" event={"ID":"44825f51-d618-4095-93fe-478a2f497eeb","Type":"ContainerDied","Data":"031f46f2cddb1dd369ba8d1fb22fd7d0c1c175912012ee473bf3feaebd3de830"}
Nov 03 20:08:36.232989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.232883    4531 scope.go:122] "RemoveContainer" containerID="0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1"
Nov 03 20:08:36.234521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.234497    4531 generic.go:356] "Generic (PLEG): container finished" podID="8306ce04-6749-4236-9359-895705d5fbe1" containerID="47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528" exitCode=0
Nov 03 20:08:36.234727 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.234711    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" event={"ID":"8306ce04-6749-4236-9359-895705d5fbe1","Type":"ContainerDied","Data":"47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528"}
Nov 03 20:08:36.234797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.234742    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz" event={"ID":"8306ce04-6749-4236-9359-895705d5fbe1","Type":"ContainerDied","Data":"9fa1e7587aee6351a04b45ca9357842230dc4f87c3733fc96991a17359f02e9a"}
Nov 03 20:08:36.234845 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.234802    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"
Nov 03 20:08:36.244580 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.244557    4531 scope.go:122] "RemoveContainer" containerID="0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1"
Nov 03 20:08:36.245035 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:36.244898    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1\": container with ID starting with 0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1 not found: ID does not exist" containerID="0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1"
Nov 03 20:08:36.245035 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.244937    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1"} err="failed to get container status \"0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1\": rpc error: code = NotFound desc = could not find container \"0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1\": container with ID starting with 0a11a7900784c543fbd385d3662114f5a4b34ff757901aa3412662c93c0375a1 not found: ID does not exist"
Nov 03 20:08:36.245035 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.244957    4531 scope.go:122] "RemoveContainer" containerID="47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528"
Nov 03 20:08:36.254079 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.254050    4531 scope.go:122] "RemoveContainer" containerID="47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528"
Nov 03 20:08:36.255454 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:36.255422    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528\": container with ID starting with 47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528 not found: ID does not exist" containerID="47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528"
Nov 03 20:08:36.255556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.255462    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528"} err="failed to get container status \"47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528\": rpc error: code = NotFound desc = could not find container \"47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528\": container with ID starting with 47a42eef0db754d487df69e5d7f97af974bdcd33b160cc71bb1863615799a528 not found: ID does not exist"
Nov 03 20:08:36.257131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.257071    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"]
Nov 03 20:08:36.259933 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.259908    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-w0-82dbbf-kserve-6b6ccf8dd5-w5zdz"]
Nov 03 20:08:36.266591 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.266566    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"]
Nov 03 20:08:36.271922 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.271897    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/simulator-w0-82dbbf-kserve-5d4768cdb7-r4qhw"]
Nov 03 20:08:36.822935 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.822822    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:36.859077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.859043    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44825f51-d618-4095-93fe-478a2f497eeb" path="/var/lib/kubelet/pods/44825f51-d618-4095-93fe-478a2f497eeb/volumes"
Nov 03 20:08:36.859528 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:36.859513    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8306ce04-6749-4236-9359-895705d5fbe1" path="/var/lib/kubelet/pods/8306ce04-6749-4236-9359-895705d5fbe1/volumes"
Nov 03 20:08:37.481162 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:37.481088    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:38.135518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:38.135477    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:38.822396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:38.822356    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:39.243502 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:39.243462    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"]
Nov 03 20:08:39.256127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:39.256092    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"]
Nov 03 20:08:39.380082 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:39.380039    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:39.524675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:39.524558    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:39.646687 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:39.646650    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:40.004090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004053    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"]
Nov 03 20:08:40.004749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004722    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main"
Nov 03 20:08:40.004749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004746    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004762    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0507af87-0ab8-46b3-9c18-45a7af463894" containerName="payload-pre-processing"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004770    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0507af87-0ab8-46b3-9c18-45a7af463894" containerName="payload-pre-processing"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004797    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9683b009-84b8-4509-b43b-1a18d6646f7c" containerName="revoke-keys"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004802    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9683b009-84b8-4509-b43b-1a18d6646f7c" containerName="revoke-keys"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004823    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004828    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004835    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004839    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004851    4531 cpu_manager.go:414] "containerMap: removing container" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004859    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004938    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8306ce04-6749-4236-9359-895705d5fbe1" containerName="main"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004948    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0507af87-0ab8-46b3-9c18-45a7af463894" containerName="payload-pre-processing"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004957    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9683b009-84b8-4509-b43b-1a18d6646f7c" containerName="revoke-keys"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004967    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004972    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e747c83-6f8d-47ac-8023-d86ce46bcf94" containerName="istio-proxy"
Nov 03 20:08:40.005057 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.004978    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="44825f51-d618-4095-93fe-478a2f497eeb" containerName="main"
Nov 03 20:08:40.008712 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.008691    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:40.017438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.017378    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"]
Nov 03 20:08:40.030093 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.030061    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"]
Nov 03 20:08:40.033930 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.033902    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a48046ae-nbts4"]
Nov 03 20:08:40.106461 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.106414    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:08:40.106665 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.106519    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:08:40.169675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.169636    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a731ee-xs65l\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:40.169875 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.169720    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6tb2\" (UniqueName: \"kubernetes.io/projected/fc749a2a-2ef9-4f38-b842-4fc8bb058929-kube-api-access-k6tb2\") pod \"maas-api-revoke-sub-29a731ee-xs65l\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:40.169875 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.169776    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a731ee-xs65l\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:40.271392 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.271287    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a731ee-xs65l\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:40.271591 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.271387    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6tb2\" (UniqueName: \"kubernetes.io/projected/fc749a2a-2ef9-4f38-b842-4fc8bb058929-kube-api-access-k6tb2\") pod \"maas-api-revoke-sub-29a731ee-xs65l\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:40.271591 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.271422    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a731ee-xs65l\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:40.271983 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.271960    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a731ee-xs65l\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:40.273813 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.273779    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a731ee-xs65l\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:40.278065 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.278032    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6tb2\" (UniqueName: \"kubernetes.io/projected/fc749a2a-2ef9-4f38-b842-4fc8bb058929-kube-api-access-k6tb2\") pod \"maas-api-revoke-sub-29a731ee-xs65l\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:40.328007 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.327963    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:40.607597 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.607554    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"]
Nov 03 20:08:40.612659 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:08:40.612613    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc749a2a_2ef9_4f38_b842_4fc8bb058929.slice/crio-d6aeb4918511f7b06ed9a6b7862d2a205ef103aae9d35be16e97b524edec014d WatchSource:0}: Error finding container d6aeb4918511f7b06ed9a6b7862d2a205ef103aae9d35be16e97b524edec014d: Status 404 returned error can't find the container with id d6aeb4918511f7b06ed9a6b7862d2a205ef103aae9d35be16e97b524edec014d
Nov 03 20:08:40.814487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.814445    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"]
Nov 03 20:08:40.814742 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.814710    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz" podUID="dea0c8fc-05ab-4173-8084-ad7e2bc678ad" containerName="maas-api" containerID="cri-o://2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221" gracePeriod=30
Nov 03 20:08:40.858421 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.858340    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811cc1b2-8239-4b7f-ab01-08eba26baed8" path="/var/lib/kubelet/pods/811cc1b2-8239-4b7f-ab01-08eba26baed8/volumes"
Nov 03 20:08:40.911831 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:40.911772    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:41.046092 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.046049    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:41.071184 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.071147    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 20:08:41.174457 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.174419    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:41.194076 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.194029    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-metrics-certs\") pod \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") "
Nov 03 20:08:41.194318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.194210    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-maas-api-tls\") pod \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") "
Nov 03 20:08:41.194318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.194275    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-kube-api-access-qhd2g\" (UniqueName: \"kubernetes.io/projected/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-kube-api-access-qhd2g\") pod \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") "
Nov 03 20:08:41.194318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.194315    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-openshift-service-ca\") pod \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\" (UID: \"dea0c8fc-05ab-4173-8084-ad7e2bc678ad\") "
Nov 03 20:08:41.194844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.194806    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-openshift-service-ca" pod "dea0c8fc-05ab-4173-8084-ad7e2bc678ad" (UID: "dea0c8fc-05ab-4173-8084-ad7e2bc678ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:41.196512 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.196479    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-kube-api-access-qhd2g" pod "dea0c8fc-05ab-4173-8084-ad7e2bc678ad" (UID: "dea0c8fc-05ab-4173-8084-ad7e2bc678ad"). InnerVolumeSpecName "kube-api-access-qhd2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:41.196512 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.196480    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-metrics-certs" pod "dea0c8fc-05ab-4173-8084-ad7e2bc678ad" (UID: "dea0c8fc-05ab-4173-8084-ad7e2bc678ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:41.196748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.196724    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-maas-api-tls" pod "dea0c8fc-05ab-4173-8084-ad7e2bc678ad" (UID: "dea0c8fc-05ab-4173-8084-ad7e2bc678ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:41.271506 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.271469    4531 generic.go:356] "Generic (PLEG): container finished" podID="dea0c8fc-05ab-4173-8084-ad7e2bc678ad" containerID="2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221" exitCode=0
Nov 03 20:08:41.271711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.271521    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"
Nov 03 20:08:41.271711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.271556    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz" event={"ID":"dea0c8fc-05ab-4173-8084-ad7e2bc678ad","Type":"ContainerDied","Data":"2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221"}
Nov 03 20:08:41.271711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.271612    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz" event={"ID":"dea0c8fc-05ab-4173-8084-ad7e2bc678ad","Type":"ContainerDied","Data":"547cc54f5d8d7adc2de88fa87e06278c9b79349998d739ed352cd090f4b0c84a"}
Nov 03 20:08:41.271711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.271652    4531 scope.go:122] "RemoveContainer" containerID="2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221"
Nov 03 20:08:41.273510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.273477    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l" event={"ID":"fc749a2a-2ef9-4f38-b842-4fc8bb058929","Type":"ContainerStarted","Data":"6e42a53bf621650b6a4ba42051bc7b019f8f0d4f4e56ef9d89dff5f6e4e53304"}
Nov 03 20:08:41.273614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.273534    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l" event={"ID":"fc749a2a-2ef9-4f38-b842-4fc8bb058929","Type":"ContainerStarted","Data":"d6aeb4918511f7b06ed9a6b7862d2a205ef103aae9d35be16e97b524edec014d"}
Nov 03 20:08:41.288336 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.288282    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l" podStartSLOduration=2.288267531 podStartE2EDuration="2.288267531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:41.287524891 +0000 UTC m=+2234.557648187" watchObservedRunningTime="2026-11-03 20:08:41.288267531 +0000 UTC m=+2234.558390832"
Nov 03 20:08:41.295625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.295596    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:41.295625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.295621    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhd2g\" (UniqueName: \"kubernetes.io/projected/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-kube-api-access-qhd2g\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:41.295833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.295632    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:41.295833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.295641    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea0c8fc-05ab-4173-8084-ad7e2bc678ad-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:41.296783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.296767    4531 scope.go:122] "RemoveContainer" containerID="2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221"
Nov 03 20:08:41.297095 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:41.297074    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221\": container with ID starting with 2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221 not found: ID does not exist" containerID="2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221"
Nov 03 20:08:41.297183 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.297105    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221"} err="failed to get container status \"2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221\": rpc error: code = NotFound desc = could not find container \"2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221\": container with ID starting with 2dc8dc59839b0c2dbf00e94619a5281efd30eeb33ba3e712b59c39dd8d5e3221 not found: ID does not exist"
Nov 03 20:08:41.305847 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.305819    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"]
Nov 03 20:08:41.308357 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.308333    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-82dbbf-85fd465889-lbjgz"]
Nov 03 20:08:41.592352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:41.592262    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:42.283472 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:42.283437    4531 generic.go:356] "Generic (PLEG): container finished" podID="fc749a2a-2ef9-4f38-b842-4fc8bb058929" containerID="6e42a53bf621650b6a4ba42051bc7b019f8f0d4f4e56ef9d89dff5f6e4e53304" exitCode=0
Nov 03 20:08:42.283898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:42.283523    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l" event={"ID":"fc749a2a-2ef9-4f38-b842-4fc8bb058929","Type":"ContainerDied","Data":"6e42a53bf621650b6a4ba42051bc7b019f8f0d4f4e56ef9d89dff5f6e4e53304"}
Nov 03 20:08:42.865276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:42.865241    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea0c8fc-05ab-4173-8084-ad7e2bc678ad" path="/var/lib/kubelet/pods/dea0c8fc-05ab-4173-8084-ad7e2bc678ad/volumes"
Nov 03 20:08:43.433848 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:43.433821    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:43.523012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:43.522965    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-openshift-ca\") pod \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") "
Nov 03 20:08:43.523242 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:43.523027    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-xks-ca\") pod \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") "
Nov 03 20:08:43.523242 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:43.523109    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc749a2a-2ef9-4f38-b842-4fc8bb058929-kube-api-access-k6tb2\" (UniqueName: \"kubernetes.io/projected/fc749a2a-2ef9-4f38-b842-4fc8bb058929-kube-api-access-k6tb2\") pod \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\" (UID: \"fc749a2a-2ef9-4f38-b842-4fc8bb058929\") "
Nov 03 20:08:43.523421 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:43.523398    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-openshift-ca" pod "fc749a2a-2ef9-4f38-b842-4fc8bb058929" (UID: "fc749a2a-2ef9-4f38-b842-4fc8bb058929"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:43.525338 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:43.525308    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-xks-ca" pod "fc749a2a-2ef9-4f38-b842-4fc8bb058929" (UID: "fc749a2a-2ef9-4f38-b842-4fc8bb058929"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:43.525449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:43.525399    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc749a2a-2ef9-4f38-b842-4fc8bb058929-kube-api-access-k6tb2" pod "fc749a2a-2ef9-4f38-b842-4fc8bb058929" (UID: "fc749a2a-2ef9-4f38-b842-4fc8bb058929"). InnerVolumeSpecName "kube-api-access-k6tb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:43.624592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:43.624494    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:43.624592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:43.624530    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6tb2\" (UniqueName: \"kubernetes.io/projected/fc749a2a-2ef9-4f38-b842-4fc8bb058929-kube-api-access-k6tb2\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:43.624592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:43.624545    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc749a2a-2ef9-4f38-b842-4fc8bb058929-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:44.244141 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:44.244095    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg" podUID="f596e70f-a49c-47e8-8b8d-e5d27612f5c7" containerName="payload-processing" containerID="cri-o://52466bd853f4d7f9c59844944a312f5f0d6ecc0d2d13715d02ff1ce0cfb1fc16" gracePeriod=25
Nov 03 20:08:44.256850 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:44.256812    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj" podUID="72c91d62-295e-497e-a507-6244fa438caa" containerName="payload-pre-processing" containerID="cri-o://02fd31d8e5afee285f86496f6e597632374cb4bdac1e4a6d27b43e1aaa511623" gracePeriod=25
Nov 03 20:08:44.297570 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:44.297540    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"
Nov 03 20:08:44.297759 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:44.297537    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l" event={"ID":"fc749a2a-2ef9-4f38-b842-4fc8bb058929","Type":"ContainerDied","Data":"d6aeb4918511f7b06ed9a6b7862d2a205ef103aae9d35be16e97b524edec014d"}
Nov 03 20:08:44.297759 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:44.297666    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6aeb4918511f7b06ed9a6b7862d2a205ef103aae9d35be16e97b524edec014d"
Nov 03 20:08:45.598330 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:45.598286    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg" podUID="f596e70f-a49c-47e8-8b8d-e5d27612f5c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:08:46.604213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:46.604153    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj" podUID="72c91d62-295e-497e-a507-6244fa438caa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:08:47.302671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:47.302628    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:47.545517 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:47.545479    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:47.588867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:47.588761    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"]
Nov 03 20:08:47.589138 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:47.589104    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy" containerID="cri-o://ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d" gracePeriod=30
Nov 03 20:08:47.695798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:47.688697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:48.948644 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:48.948576    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main" containerID="cri-o://34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2" gracePeriod=45
Nov 03 20:08:48.953526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:48.953422    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main" containerID="cri-o://42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082" gracePeriod=45
Nov 03 20:08:48.975075 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:48.975039    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main" containerID="cri-o://0850ac68e9b5439e0ffef86e46c46ae13c412829c02ca3a9728d47abb1d8cfce" gracePeriod=45
Nov 03 20:08:49.236581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.236538    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main" containerID="cri-o://1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0" gracePeriod=45
Nov 03 20:08:49.239074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.239046    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 20:08:49.252625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.252600    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 20:08:49.285593 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285546    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fc563f-e643-452b-91bd-fee9d1e8e739-kube-api-access-sqmbx\" (UniqueName: \"kubernetes.io/projected/c0fc563f-e643-452b-91bd-fee9d1e8e739-kube-api-access-sqmbx\") pod \"c0fc563f-e643-452b-91bd-fee9d1e8e739\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") "
Nov 03 20:08:49.285785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285613    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-home\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-home\") pod \"c0fc563f-e643-452b-91bd-fee9d1e8e739\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") "
Nov 03 20:08:49.285785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285648    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-model-cache\") pod \"e72488d0-dae8-4b0f-9c8c-9695319069b5\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") "
Nov 03 20:08:49.285785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285674    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-dshm\") pod \"c0fc563f-e643-452b-91bd-fee9d1e8e739\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") "
Nov 03 20:08:49.285785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285713    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0fc563f-e643-452b-91bd-fee9d1e8e739-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0fc563f-e643-452b-91bd-fee9d1e8e739-tls-certs\") pod \"c0fc563f-e643-452b-91bd-fee9d1e8e739\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") "
Nov 03 20:08:49.285785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285738    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-home\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-home\") pod \"e72488d0-dae8-4b0f-9c8c-9695319069b5\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") "
Nov 03 20:08:49.286090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285796    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-model-cache\") pod \"c0fc563f-e643-452b-91bd-fee9d1e8e739\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") "
Nov 03 20:08:49.286090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285858    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72488d0-dae8-4b0f-9c8c-9695319069b5-kube-api-access-24bpn\" (UniqueName: \"kubernetes.io/projected/e72488d0-dae8-4b0f-9c8c-9695319069b5-kube-api-access-24bpn\") pod \"e72488d0-dae8-4b0f-9c8c-9695319069b5\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") "
Nov 03 20:08:49.286090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285896    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e72488d0-dae8-4b0f-9c8c-9695319069b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/e72488d0-dae8-4b0f-9c8c-9695319069b5-tls-certs\") pod \"e72488d0-dae8-4b0f-9c8c-9695319069b5\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") "
Nov 03 20:08:49.286090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285925    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-tmp-dir\") pod \"e72488d0-dae8-4b0f-9c8c-9695319069b5\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") "
Nov 03 20:08:49.286090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285975    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-tmp-dir\") pod \"c0fc563f-e643-452b-91bd-fee9d1e8e739\" (UID: \"c0fc563f-e643-452b-91bd-fee9d1e8e739\") "
Nov 03 20:08:49.286090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285975    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-model-cache" pod "e72488d0-dae8-4b0f-9c8c-9695319069b5" (UID: "e72488d0-dae8-4b0f-9c8c-9695319069b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.286090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.285982    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-home" pod "c0fc563f-e643-452b-91bd-fee9d1e8e739" (UID: "c0fc563f-e643-452b-91bd-fee9d1e8e739"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.286090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.286065    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-dshm\") pod \"e72488d0-dae8-4b0f-9c8c-9695319069b5\" (UID: \"e72488d0-dae8-4b0f-9c8c-9695319069b5\") "
Nov 03 20:08:49.286545 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.286136    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-model-cache" pod "c0fc563f-e643-452b-91bd-fee9d1e8e739" (UID: "c0fc563f-e643-452b-91bd-fee9d1e8e739"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.286545 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.286245    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-home" pod "e72488d0-dae8-4b0f-9c8c-9695319069b5" (UID: "e72488d0-dae8-4b0f-9c8c-9695319069b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.286668 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.286565    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.286668 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.286589    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.286668 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.286605    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.286668 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.286620    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.286893 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.286744    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-tmp-dir" pod "e72488d0-dae8-4b0f-9c8c-9695319069b5" (UID: "e72488d0-dae8-4b0f-9c8c-9695319069b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.286893 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.286869    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-tmp-dir" pod "c0fc563f-e643-452b-91bd-fee9d1e8e739" (UID: "c0fc563f-e643-452b-91bd-fee9d1e8e739"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.288535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.288512    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fc563f-e643-452b-91bd-fee9d1e8e739-kube-api-access-sqmbx" pod "c0fc563f-e643-452b-91bd-fee9d1e8e739" (UID: "c0fc563f-e643-452b-91bd-fee9d1e8e739"). InnerVolumeSpecName "kube-api-access-sqmbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:49.288663 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.288580    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72488d0-dae8-4b0f-9c8c-9695319069b5-tls-certs" pod "e72488d0-dae8-4b0f-9c8c-9695319069b5" (UID: "e72488d0-dae8-4b0f-9c8c-9695319069b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:49.288868 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.288848    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-dshm" pod "e72488d0-dae8-4b0f-9c8c-9695319069b5" (UID: "e72488d0-dae8-4b0f-9c8c-9695319069b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.289071 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.289050    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-dshm" pod "c0fc563f-e643-452b-91bd-fee9d1e8e739" (UID: "c0fc563f-e643-452b-91bd-fee9d1e8e739"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.289721 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.289696    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fc563f-e643-452b-91bd-fee9d1e8e739-tls-certs" pod "c0fc563f-e643-452b-91bd-fee9d1e8e739" (UID: "c0fc563f-e643-452b-91bd-fee9d1e8e739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:49.289816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.289729    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72488d0-dae8-4b0f-9c8c-9695319069b5-kube-api-access-24bpn" pod "e72488d0-dae8-4b0f-9c8c-9695319069b5" (UID: "e72488d0-dae8-4b0f-9c8c-9695319069b5"). InnerVolumeSpecName "kube-api-access-24bpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:49.333792 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.333754    4531 generic.go:356] "Generic (PLEG): container finished" podID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerID="0850ac68e9b5439e0ffef86e46c46ae13c412829c02ca3a9728d47abb1d8cfce" exitCode=0
Nov 03 20:08:49.333954 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.333820    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" event={"ID":"c60b85d3-ab1a-482a-86f0-32ce873c7383","Type":"ContainerDied","Data":"0850ac68e9b5439e0ffef86e46c46ae13c412829c02ca3a9728d47abb1d8cfce"}
Nov 03 20:08:49.335442 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.335418    4531 generic.go:356] "Generic (PLEG): container finished" podID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerID="42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082" exitCode=0
Nov 03 20:08:49.335574 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.335473    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"
Nov 03 20:08:49.335574 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.335507    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" event={"ID":"e72488d0-dae8-4b0f-9c8c-9695319069b5","Type":"ContainerDied","Data":"42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082"}
Nov 03 20:08:49.335574 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.335549    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f" event={"ID":"e72488d0-dae8-4b0f-9c8c-9695319069b5","Type":"ContainerDied","Data":"bcdeef383da8593a0765fbf6345f3a1bdfe1af2be87abc0772a68823f0b19b4a"}
Nov 03 20:08:49.335574 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.335572    4531 scope.go:122] "RemoveContainer" containerID="42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082"
Nov 03 20:08:49.337037 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.337014    4531 generic.go:356] "Generic (PLEG): container finished" podID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerID="34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2" exitCode=0
Nov 03 20:08:49.337148 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.337057    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"
Nov 03 20:08:49.337148 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.337103    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" event={"ID":"c0fc563f-e643-452b-91bd-fee9d1e8e739","Type":"ContainerDied","Data":"34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2"}
Nov 03 20:08:49.337148 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.337145    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf" event={"ID":"c0fc563f-e643-452b-91bd-fee9d1e8e739","Type":"ContainerDied","Data":"026680fcec3683cdf8fd4fe4afdf88e72cc9ea30026008a715394b65471a08ef"}
Nov 03 20:08:49.354916 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.354897    4531 scope.go:122] "RemoveContainer" containerID="42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082"
Nov 03 20:08:49.356136 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:49.356107    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082\": container with ID starting with 42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082 not found: ID does not exist" containerID="42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082"
Nov 03 20:08:49.356367 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.356340    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082"} err="failed to get container status \"42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082\": rpc error: code = NotFound desc = could not find container \"42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082\": container with ID starting with 42cf903a3877422929f333a2150af386ebf580e42176f85cbcdbdc4edef95082 not found: ID does not exist"
Nov 03 20:08:49.356462 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.356452    4531 scope.go:122] "RemoveContainer" containerID="34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2"
Nov 03 20:08:49.360780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.360748    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 20:08:49.366168 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.366141    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"]
Nov 03 20:08:49.369240 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.369218    4531 scope.go:122] "RemoveContainer" containerID="34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2"
Nov 03 20:08:49.369608 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:49.369586    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2\": container with ID starting with 34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2 not found: ID does not exist" containerID="34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2"
Nov 03 20:08:49.369681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.369615    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2"} err="failed to get container status \"34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2\": rpc error: code = NotFound desc = could not find container \"34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2\": container with ID starting with 34b4559ca58543b56324e065efcc6650e12d4a556dfb30825c7cf89815ca63f2 not found: ID does not exist"
Nov 03 20:08:49.370096 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.370075    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-2-w5-0486e8-kserve-6bf7dcb44d-g9jnf"]
Nov 03 20:08:49.379779 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.379746    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"]
Nov 03 20:08:49.382319 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.382288    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/distinct-w5-0486e8-kserve-76d9c7596c-k9w5f"]
Nov 03 20:08:49.387094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387066    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-home\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-home\") pod \"c60b85d3-ab1a-482a-86f0-32ce873c7383\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") "
Nov 03 20:08:49.387278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387115    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60b85d3-ab1a-482a-86f0-32ce873c7383-kube-api-access-4gqkp\" (UniqueName: \"kubernetes.io/projected/c60b85d3-ab1a-482a-86f0-32ce873c7383-kube-api-access-4gqkp\") pod \"c60b85d3-ab1a-482a-86f0-32ce873c7383\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") "
Nov 03 20:08:49.387278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387136    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c60b85d3-ab1a-482a-86f0-32ce873c7383-tls-certs\" (UniqueName: \"kubernetes.io/secret/c60b85d3-ab1a-482a-86f0-32ce873c7383-tls-certs\") pod \"c60b85d3-ab1a-482a-86f0-32ce873c7383\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") "
Nov 03 20:08:49.387278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387168    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-dshm\") pod \"c60b85d3-ab1a-482a-86f0-32ce873c7383\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") "
Nov 03 20:08:49.387278 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387240    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-tmp-dir\") pod \"c60b85d3-ab1a-482a-86f0-32ce873c7383\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") "
Nov 03 20:08:49.387562 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387304    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-model-cache\") pod \"c60b85d3-ab1a-482a-86f0-32ce873c7383\" (UID: \"c60b85d3-ab1a-482a-86f0-32ce873c7383\") "
Nov 03 20:08:49.387562 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387373    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-home" pod "c60b85d3-ab1a-482a-86f0-32ce873c7383" (UID: "c60b85d3-ab1a-482a-86f0-32ce873c7383"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.387684 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387546    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-tmp-dir" pod "c60b85d3-ab1a-482a-86f0-32ce873c7383" (UID: "c60b85d3-ab1a-482a-86f0-32ce873c7383"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.387684 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387612    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-model-cache" pod "c60b85d3-ab1a-482a-86f0-32ce873c7383" (UID: "c60b85d3-ab1a-482a-86f0-32ce873c7383"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.387797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387717    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.387797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387736    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24bpn\" (UniqueName: \"kubernetes.io/projected/e72488d0-dae8-4b0f-9c8c-9695319069b5-kube-api-access-24bpn\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.387797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387746    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e72488d0-dae8-4b0f-9c8c-9695319069b5-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.387797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387773    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.387797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387781    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.387797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387790    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.387797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387798    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e72488d0-dae8-4b0f-9c8c-9695319069b5-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.388129 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387806    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.388129 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387814    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqmbx\" (UniqueName: \"kubernetes.io/projected/c0fc563f-e643-452b-91bd-fee9d1e8e739-kube-api-access-sqmbx\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.388129 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387824    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0fc563f-e643-452b-91bd-fee9d1e8e739-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.388129 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.387832    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0fc563f-e643-452b-91bd-fee9d1e8e739-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.389355 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.389329    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60b85d3-ab1a-482a-86f0-32ce873c7383-kube-api-access-4gqkp" pod "c60b85d3-ab1a-482a-86f0-32ce873c7383" (UID: "c60b85d3-ab1a-482a-86f0-32ce873c7383"). InnerVolumeSpecName "kube-api-access-4gqkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:49.389779 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.389758    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60b85d3-ab1a-482a-86f0-32ce873c7383-tls-certs" pod "c60b85d3-ab1a-482a-86f0-32ce873c7383" (UID: "c60b85d3-ab1a-482a-86f0-32ce873c7383"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:49.389861 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.389783    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-dshm" pod "c60b85d3-ab1a-482a-86f0-32ce873c7383" (UID: "c60b85d3-ab1a-482a-86f0-32ce873c7383"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.489126 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.489033    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gqkp\" (UniqueName: \"kubernetes.io/projected/c60b85d3-ab1a-482a-86f0-32ce873c7383-kube-api-access-4gqkp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.489126 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.489068    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c60b85d3-ab1a-482a-86f0-32ce873c7383-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.489126 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.489081    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c60b85d3-ab1a-482a-86f0-32ce873c7383-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.520156 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.520127    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 20:08:49.589591 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.589556    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-dshm\") pod \"d4bc662c-d375-4d6a-83eb-32481c8f7902\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") "
Nov 03 20:08:49.589591 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.589596    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-home\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-home\") pod \"d4bc662c-d375-4d6a-83eb-32481c8f7902\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") "
Nov 03 20:08:49.589834 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.589625    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-model-cache\") pod \"d4bc662c-d375-4d6a-83eb-32481c8f7902\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") "
Nov 03 20:08:49.589834 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.589729    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-tmp-dir\") pod \"d4bc662c-d375-4d6a-83eb-32481c8f7902\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") "
Nov 03 20:08:49.589834 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.589808    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4bc662c-d375-4d6a-83eb-32481c8f7902-kube-api-access-9nqpp\" (UniqueName: \"kubernetes.io/projected/d4bc662c-d375-4d6a-83eb-32481c8f7902-kube-api-access-9nqpp\") pod \"d4bc662c-d375-4d6a-83eb-32481c8f7902\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") "
Nov 03 20:08:49.590012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.589840    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4bc662c-d375-4d6a-83eb-32481c8f7902-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bc662c-d375-4d6a-83eb-32481c8f7902-tls-certs\") pod \"d4bc662c-d375-4d6a-83eb-32481c8f7902\" (UID: \"d4bc662c-d375-4d6a-83eb-32481c8f7902\") "
Nov 03 20:08:49.590012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.589919    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-home" pod "d4bc662c-d375-4d6a-83eb-32481c8f7902" (UID: "d4bc662c-d375-4d6a-83eb-32481c8f7902"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.590012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.589941    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-model-cache" pod "d4bc662c-d375-4d6a-83eb-32481c8f7902" (UID: "d4bc662c-d375-4d6a-83eb-32481c8f7902"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.590191 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.590056    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-tmp-dir" pod "d4bc662c-d375-4d6a-83eb-32481c8f7902" (UID: "d4bc662c-d375-4d6a-83eb-32481c8f7902"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.590291 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.590273    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.590355 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.590296    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.590355 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.590312    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.591827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.591799    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-dshm" pod "d4bc662c-d375-4d6a-83eb-32481c8f7902" (UID: "d4bc662c-d375-4d6a-83eb-32481c8f7902"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:49.591970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.591949    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4bc662c-d375-4d6a-83eb-32481c8f7902-kube-api-access-9nqpp" pod "d4bc662c-d375-4d6a-83eb-32481c8f7902" (UID: "d4bc662c-d375-4d6a-83eb-32481c8f7902"). InnerVolumeSpecName "kube-api-access-9nqpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:49.592101 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.592086    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4bc662c-d375-4d6a-83eb-32481c8f7902-tls-certs" pod "d4bc662c-d375-4d6a-83eb-32481c8f7902" (UID: "d4bc662c-d375-4d6a-83eb-32481c8f7902"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:49.691551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.691501    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bc662c-d375-4d6a-83eb-32481c8f7902-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.691551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.691539    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nqpp\" (UniqueName: \"kubernetes.io/projected/d4bc662c-d375-4d6a-83eb-32481c8f7902-kube-api-access-9nqpp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:49.691551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:49.691550    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bc662c-d375-4d6a-83eb-32481c8f7902-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:50.345620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.345524    4531 generic.go:356] "Generic (PLEG): container finished" podID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerID="1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0" exitCode=0
Nov 03 20:08:50.345620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.345574    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"
Nov 03 20:08:50.346312 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.345615    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" event={"ID":"d4bc662c-d375-4d6a-83eb-32481c8f7902","Type":"ContainerDied","Data":"1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0"}
Nov 03 20:08:50.346312 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.345667    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml" event={"ID":"d4bc662c-d375-4d6a-83eb-32481c8f7902","Type":"ContainerDied","Data":"6c9e9d9cb95a0064509833ca8ba04e96736bc60fff949f89fd8d6c6cdc0f8929"}
Nov 03 20:08:50.346312 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.345687    4531 scope.go:122] "RemoveContainer" containerID="1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0"
Nov 03 20:08:50.347234 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.347208    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk" event={"ID":"c60b85d3-ab1a-482a-86f0-32ce873c7383","Type":"ContainerDied","Data":"fe2235e494baa38af00832f5e2a1a7d2600f2e01c78a230fd72fe5fa0e48412f"}
Nov 03 20:08:50.347350 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.347214    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"
Nov 03 20:08:50.357969 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.357945    4531 scope.go:122] "RemoveContainer" containerID="1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0"
Nov 03 20:08:50.358359 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:50.358271    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0\": container with ID starting with 1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0 not found: ID does not exist" containerID="1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0"
Nov 03 20:08:50.358359 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.358300    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0"} err="failed to get container status \"1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0\": rpc error: code = NotFound desc = could not find container \"1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0\": container with ID starting with 1c02552d7a0760993fbbbf7fbbd7146390db596f875808633dde03294c5a7ec0 not found: ID does not exist"
Nov 03 20:08:50.358359 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.358321    4531 scope.go:122] "RemoveContainer" containerID="0850ac68e9b5439e0ffef86e46c46ae13c412829c02ca3a9728d47abb1d8cfce"
Nov 03 20:08:50.371711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.371687    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"]
Nov 03 20:08:50.376219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.376195    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/premium-w5-0486e8-kserve-576798c84d-fnhwk"]
Nov 03 20:08:50.383863 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.383828    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"]
Nov 03 20:08:50.387340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.387314    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0486e8/simulator-w5-0486e8-kserve-798bd88bbf-kf4ml"]
Nov 03 20:08:50.572813 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:50.572773    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:08:50.857711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.857675    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" path="/var/lib/kubelet/pods/c0fc563f-e643-452b-91bd-fee9d1e8e739/volumes"
Nov 03 20:08:50.858159 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.858144    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" path="/var/lib/kubelet/pods/c60b85d3-ab1a-482a-86f0-32ce873c7383/volumes"
Nov 03 20:08:50.858587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.858573    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" path="/var/lib/kubelet/pods/d4bc662c-d375-4d6a-83eb-32481c8f7902/volumes"
Nov 03 20:08:50.859306 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:50.859287    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" path="/var/lib/kubelet/pods/e72488d0-dae8-4b0f-9c8c-9695319069b5/volumes"
Nov 03 20:08:51.368778 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:51.368746    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:51.512133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:51.512103    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:51.645233 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:51.645167    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:51.784819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:51.784772    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:08:51.785002 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:51.784839    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:08:51.950301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:51.950216    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:52.847493 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.847468    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 20:08:52.923487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.923450    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-podinfo\") pod \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") "
Nov 03 20:08:52.923697 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.923539    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-envoy\") pod \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") "
Nov 03 20:08:52.923697 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.923561    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-certs\") pod \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") "
Nov 03 20:08:52.923697 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.923629    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-socket\") pod \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") "
Nov 03 20:08:52.923871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.923731    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-token\" (UniqueName: \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-token\") pod \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") "
Nov 03 20:08:52.923871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.923760    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-kube-api-access-k7dhg\" (UniqueName: \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-kube-api-access-k7dhg\") pod \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") "
Nov 03 20:08:52.923871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.923830    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istiod-ca-cert\") pod \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") "
Nov 03 20:08:52.923871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.923849    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-certs" pod "aa2ffbba-e334-4eb9-b86b-184fc8c04bed" (UID: "aa2ffbba-e334-4eb9-b86b-184fc8c04bed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:52.924214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.923874    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-credential-socket\") pod \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") "
Nov 03 20:08:52.924214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.923938    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-data\") pod \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\" (UID: \"aa2ffbba-e334-4eb9-b86b-184fc8c04bed\") "
Nov 03 20:08:52.924214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.923946    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-socket" pod "aa2ffbba-e334-4eb9-b86b-184fc8c04bed" (UID: "aa2ffbba-e334-4eb9-b86b-184fc8c04bed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:52.924396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.924227    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istiod-ca-cert" pod "aa2ffbba-e334-4eb9-b86b-184fc8c04bed" (UID: "aa2ffbba-e334-4eb9-b86b-184fc8c04bed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:52.924396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.924353    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:52.924396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.924375    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:52.924396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.924376    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-data" pod "aa2ffbba-e334-4eb9-b86b-184fc8c04bed" (UID: "aa2ffbba-e334-4eb9-b86b-184fc8c04bed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:52.924396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.924389    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:52.924616 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.924452    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-credential-socket" pod "aa2ffbba-e334-4eb9-b86b-184fc8c04bed" (UID: "aa2ffbba-e334-4eb9-b86b-184fc8c04bed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:52.925919 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.925894    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-kube-api-access-k7dhg" pod "aa2ffbba-e334-4eb9-b86b-184fc8c04bed" (UID: "aa2ffbba-e334-4eb9-b86b-184fc8c04bed"). InnerVolumeSpecName "kube-api-access-k7dhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:52.926022 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.925939    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-podinfo" pod "aa2ffbba-e334-4eb9-b86b-184fc8c04bed" (UID: "aa2ffbba-e334-4eb9-b86b-184fc8c04bed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:08:52.926248 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.926223    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-envoy" pod "aa2ffbba-e334-4eb9-b86b-184fc8c04bed" (UID: "aa2ffbba-e334-4eb9-b86b-184fc8c04bed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:52.926325 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:52.926230    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-token" pod "aa2ffbba-e334-4eb9-b86b-184fc8c04bed" (UID: "aa2ffbba-e334-4eb9-b86b-184fc8c04bed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:53.025689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.025648    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.025689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.025689    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.025910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.025707    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7dhg\" (UniqueName: \"kubernetes.io/projected/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-kube-api-access-k7dhg\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.025910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.025723    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.025910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.025737    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.025910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.025750    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2ffbba-e334-4eb9-b86b-184fc8c04bed-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.373929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.373843    4531 generic.go:356] "Generic (PLEG): container finished" podID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerID="ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d" exitCode=0
Nov 03 20:08:53.373929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.373901    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"
Nov 03 20:08:53.374121 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.373924    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" event={"ID":"aa2ffbba-e334-4eb9-b86b-184fc8c04bed","Type":"ContainerDied","Data":"ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d"}
Nov 03 20:08:53.374121 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.373984    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf" event={"ID":"aa2ffbba-e334-4eb9-b86b-184fc8c04bed","Type":"ContainerDied","Data":"5c6c83fee64dd39c3d52b8a470cc85687fdc4602ca9b7a5e171c1bafc234b292"}
Nov 03 20:08:53.374121 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.374009    4531 scope.go:122] "RemoveContainer" containerID="ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d"
Nov 03 20:08:53.385106 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.385079    4531 scope.go:122] "RemoveContainer" containerID="8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4"
Nov 03 20:08:53.395872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.395840    4531 scope.go:122] "RemoveContainer" containerID="ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d"
Nov 03 20:08:53.396252 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:53.396223    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d\": container with ID starting with ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d not found: ID does not exist" containerID="ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d"
Nov 03 20:08:53.396363 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.396262    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d"} err="failed to get container status \"ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d\": rpc error: code = NotFound desc = could not find container \"ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d\": container with ID starting with ca80d944106465933ac5ec621975a3319b4b9d7ffa13647fab7a06fe94a5180d not found: ID does not exist"
Nov 03 20:08:53.396363 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.396285    4531 scope.go:122] "RemoveContainer" containerID="8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4"
Nov 03 20:08:53.396610 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:08:53.396589    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4\": container with ID starting with 8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4 not found: ID does not exist" containerID="8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4"
Nov 03 20:08:53.396674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.396616    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4"} err="failed to get container status \"8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4\": rpc error: code = NotFound desc = could not find container \"8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4\": container with ID starting with 8d623def70540470ded4e10cbf1b995bfc4a21875458e9300e096b1c18c2b8d4 not found: ID does not exist"
Nov 03 20:08:53.398145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.398121    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"]
Nov 03 20:08:53.404526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:53.404500    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-cfqjf"]
Nov 03 20:08:54.858184 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:54.858146    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" path="/var/lib/kubelet/pods/aa2ffbba-e334-4eb9-b86b-184fc8c04bed/volumes"
Nov 03 20:08:55.598165 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:55.598125    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg" podUID="f596e70f-a49c-47e8-8b8d-e5d27612f5c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:08:56.604514 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:56.604476    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj" podUID="72c91d62-295e-497e-a507-6244fa438caa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:08:57.092397 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:57.092340    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:57.216852 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:57.216816    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:58.256102 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:58.256060    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:58.542429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:58.542340    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:58.680983 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:58.680948    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:58.779397 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:58.779337    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"]
Nov 03 20:08:58.780412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:58.780374    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" containerID="cri-o://d5c899d07753b75ea3fc9beab48520f756fa99113df7f362a333224a1297f2be" gracePeriod=30
Nov 03 20:08:58.859163 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:58.859057    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:59.375875 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:59.375833    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:08:59.376305 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:08:59.375927    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:09:03.105006 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.104958    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86c547b4b-wcxhd"]
Nov 03 20:09:03.105674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105633    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main"
Nov 03 20:09:03.105674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105661    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105690    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105699    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105718    4531 cpu_manager.go:414] "containerMap: removing container" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105727    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105739    4531 cpu_manager.go:414] "containerMap: removing container" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105746    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105759    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dea0c8fc-05ab-4173-8084-ad7e2bc678ad" containerName="maas-api"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105767    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea0c8fc-05ab-4173-8084-ad7e2bc678ad" containerName="maas-api"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105777    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105784    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105795    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fc749a2a-2ef9-4f38-b842-4fc8bb058929" containerName="revoke-keys"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105803    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc749a2a-2ef9-4f38-b842-4fc8bb058929" containerName="revoke-keys"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105823    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main"
Nov 03 20:09:03.105837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105831    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main"
Nov 03 20:09:03.106679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105948    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4bc662c-d375-4d6a-83eb-32481c8f7902" containerName="main"
Nov 03 20:09:03.106679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105962    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fc563f-e643-452b-91bd-fee9d1e8e739" containerName="main"
Nov 03 20:09:03.106679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105972    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy"
Nov 03 20:09:03.106679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105983    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2ffbba-e334-4eb9-b86b-184fc8c04bed" containerName="istio-proxy"
Nov 03 20:09:03.106679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.105994    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea0c8fc-05ab-4173-8084-ad7e2bc678ad" containerName="maas-api"
Nov 03 20:09:03.106679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.106004    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72488d0-dae8-4b0f-9c8c-9695319069b5" containerName="main"
Nov 03 20:09:03.106679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.106013    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc749a2a-2ef9-4f38-b842-4fc8bb058929" containerName="revoke-keys"
Nov 03 20:09:03.106679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.106023    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60b85d3-ab1a-482a-86f0-32ce873c7383" containerName="main"
Nov 03 20:09:03.111369 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.111342    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:03.116275 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.116245    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 20:09:03.116430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.116290    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 20:09:03.116430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.116319    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5cw4d\""
Nov 03 20:09:03.116430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.116373    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 20:09:03.117826 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.117800    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86c547b4b-wcxhd"]
Nov 03 20:09:03.239774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.239731    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-metrics-certs\") pod \"maas-controller-86c547b4b-wcxhd\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") " pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:03.239774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.239774    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnlz8\" (UniqueName: \"kubernetes.io/projected/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-kube-api-access-dnlz8\") pod \"maas-controller-86c547b4b-wcxhd\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") " pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:03.240085 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.240040    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-webhook-cert\") pod \"maas-controller-86c547b4b-wcxhd\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") " pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:03.341535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.341470    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-webhook-cert\") pod \"maas-controller-86c547b4b-wcxhd\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") " pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:03.341745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.341582    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-metrics-certs\") pod \"maas-controller-86c547b4b-wcxhd\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") " pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:03.341745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.341615    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnlz8\" (UniqueName: \"kubernetes.io/projected/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-kube-api-access-dnlz8\") pod \"maas-controller-86c547b4b-wcxhd\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") " pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:03.344981 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.344950    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-metrics-certs\") pod \"maas-controller-86c547b4b-wcxhd\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") " pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:03.345151 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.345124    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-webhook-cert\") pod \"maas-controller-86c547b4b-wcxhd\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") " pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:03.349626 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.349595    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnlz8\" (UniqueName: \"kubernetes.io/projected/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-kube-api-access-dnlz8\") pod \"maas-controller-86c547b4b-wcxhd\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") " pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:03.436200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.435387    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:03.835510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.835481    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86c547b4b-wcxhd"]
Nov 03 20:09:03.897566 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:03.879400    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:09:04.091915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.088096    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:04.529817 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.529788    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c547b4b-wcxhd" event={"ID":"cb11cb0b-8af0-4643-95eb-1318dcaf0d13","Type":"ContainerStarted","Data":"a1d0803c7ce5cedbcd83b6270ea0c82c63fe65323b27783d9979e32c30741430"}
Nov 03 20:09:04.532525 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.532495    4531 generic.go:356] "Generic (PLEG): container finished" podID="72c91d62-295e-497e-a507-6244fa438caa" containerID="02fd31d8e5afee285f86496f6e597632374cb4bdac1e4a6d27b43e1aaa511623" exitCode=0
Nov 03 20:09:04.532684 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.532624    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj" event={"ID":"72c91d62-295e-497e-a507-6244fa438caa","Type":"ContainerDied","Data":"02fd31d8e5afee285f86496f6e597632374cb4bdac1e4a6d27b43e1aaa511623"}
Nov 03 20:09:04.535411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.534759    4531 generic.go:356] "Generic (PLEG): container finished" podID="f9542796-9bc6-4afc-b664-35a17397bc61" containerID="d5c899d07753b75ea3fc9beab48520f756fa99113df7f362a333224a1297f2be" exitCode=0
Nov 03 20:09:04.535411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.534852    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" event={"ID":"f9542796-9bc6-4afc-b664-35a17397bc61","Type":"ContainerDied","Data":"d5c899d07753b75ea3fc9beab48520f756fa99113df7f362a333224a1297f2be"}
Nov 03 20:09:04.547409 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.546458    4531 generic.go:356] "Generic (PLEG): container finished" podID="f596e70f-a49c-47e8-8b8d-e5d27612f5c7" containerID="52466bd853f4d7f9c59844944a312f5f0d6ecc0d2d13715d02ff1ce0cfb1fc16" exitCode=0
Nov 03 20:09:04.547409 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.546680    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg" event={"ID":"f596e70f-a49c-47e8-8b8d-e5d27612f5c7","Type":"ContainerDied","Data":"52466bd853f4d7f9c59844944a312f5f0d6ecc0d2d13715d02ff1ce0cfb1fc16"}
Nov 03 20:09:04.849286 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.841543    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 20:09:04.938911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.937016    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9542796-9bc6-4afc-b664-35a17397bc61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9542796-9bc6-4afc-b664-35a17397bc61-istio-podinfo\") pod \"f9542796-9bc6-4afc-b664-35a17397bc61\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") "
Nov 03 20:09:04.938911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.937073    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-data\") pod \"f9542796-9bc6-4afc-b664-35a17397bc61\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") "
Nov 03 20:09:04.938911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.937125    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-kube-api-access-qjzxc\" (UniqueName: \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-kube-api-access-qjzxc\") pod \"f9542796-9bc6-4afc-b664-35a17397bc61\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") "
Nov 03 20:09:04.938911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.937164    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-credential-socket\") pod \"f9542796-9bc6-4afc-b664-35a17397bc61\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") "
Nov 03 20:09:04.938911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.937226    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-istio-token\" (UniqueName: \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-istio-token\") pod \"f9542796-9bc6-4afc-b664-35a17397bc61\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") "
Nov 03 20:09:04.938911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.937251    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-certs\") pod \"f9542796-9bc6-4afc-b664-35a17397bc61\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") "
Nov 03 20:09:04.938911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.937324    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-socket\") pod \"f9542796-9bc6-4afc-b664-35a17397bc61\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") "
Nov 03 20:09:04.938911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.937385    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-envoy\") pod \"f9542796-9bc6-4afc-b664-35a17397bc61\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") "
Nov 03 20:09:04.938911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.937463    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9542796-9bc6-4afc-b664-35a17397bc61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9542796-9bc6-4afc-b664-35a17397bc61-istiod-ca-cert\") pod \"f9542796-9bc6-4afc-b664-35a17397bc61\" (UID: \"f9542796-9bc6-4afc-b664-35a17397bc61\") "
Nov 03 20:09:04.945921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.942555    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9542796-9bc6-4afc-b664-35a17397bc61-istiod-ca-cert" pod "f9542796-9bc6-4afc-b664-35a17397bc61" (UID: "f9542796-9bc6-4afc-b664-35a17397bc61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:09:04.945921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.942858    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-socket" pod "f9542796-9bc6-4afc-b664-35a17397bc61" (UID: "f9542796-9bc6-4afc-b664-35a17397bc61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:04.945921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.943784    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-credential-socket" pod "f9542796-9bc6-4afc-b664-35a17397bc61" (UID: "f9542796-9bc6-4afc-b664-35a17397bc61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:04.945921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.944086    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-data" pod "f9542796-9bc6-4afc-b664-35a17397bc61" (UID: "f9542796-9bc6-4afc-b664-35a17397bc61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:04.945921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.945883    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-certs" pod "f9542796-9bc6-4afc-b664-35a17397bc61" (UID: "f9542796-9bc6-4afc-b664-35a17397bc61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:04.972654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.972547    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-envoy" pod "f9542796-9bc6-4afc-b664-35a17397bc61" (UID: "f9542796-9bc6-4afc-b664-35a17397bc61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:04.972982 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.972938    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9542796-9bc6-4afc-b664-35a17397bc61-istio-podinfo" pod "f9542796-9bc6-4afc-b664-35a17397bc61" (UID: "f9542796-9bc6-4afc-b664-35a17397bc61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:09:05.006752 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:04.985477    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-kube-api-access-qjzxc" pod "f9542796-9bc6-4afc-b664-35a17397bc61" (UID: "f9542796-9bc6-4afc-b664-35a17397bc61"). InnerVolumeSpecName "kube-api-access-qjzxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:05.028297 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.028247    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-istio-token" pod "f9542796-9bc6-4afc-b664-35a17397bc61" (UID: "f9542796-9bc6-4afc-b664-35a17397bc61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:05.034910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.034877    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 20:09:05.040927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.040700    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9542796-9bc6-4afc-b664-35a17397bc61-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.040927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.040735    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9542796-9bc6-4afc-b664-35a17397bc61-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.040927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.040749    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.040927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.040763    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjzxc\" (UniqueName: \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-kube-api-access-qjzxc\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.040927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.040780    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.040927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.040793    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9542796-9bc6-4afc-b664-35a17397bc61-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.040927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.040806    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.040927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.040820    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.040927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.040834    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9542796-9bc6-4afc-b664-35a17397bc61-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.111967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.111936    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 20:09:05.141865 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.141827    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-plugins-config-volume\") pod \"f596e70f-a49c-47e8-8b8d-e5d27612f5c7\" (UID: \"f596e70f-a49c-47e8-8b8d-e5d27612f5c7\") "
Nov 03 20:09:05.142955 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.142622    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-plugins-config-volume" pod "f596e70f-a49c-47e8-8b8d-e5d27612f5c7" (UID: "f596e70f-a49c-47e8-8b8d-e5d27612f5c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:09:05.143216 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.143195    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-kube-api-access-xvghb\" (UniqueName: \"kubernetes.io/projected/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-kube-api-access-xvghb\") pod \"f596e70f-a49c-47e8-8b8d-e5d27612f5c7\" (UID: \"f596e70f-a49c-47e8-8b8d-e5d27612f5c7\") "
Nov 03 20:09:05.143735 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.143717    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c91d62-295e-497e-a507-6244fa438caa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c91d62-295e-497e-a507-6244fa438caa-plugins-config-volume\") pod \"72c91d62-295e-497e-a507-6244fa438caa\" (UID: \"72c91d62-295e-497e-a507-6244fa438caa\") "
Nov 03 20:09:05.144513 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.144494    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c91d62-295e-497e-a507-6244fa438caa-kube-api-access-6g6jw\" (UniqueName: \"kubernetes.io/projected/72c91d62-295e-497e-a507-6244fa438caa-kube-api-access-6g6jw\") pod \"72c91d62-295e-497e-a507-6244fa438caa\" (UID: \"72c91d62-295e-497e-a507-6244fa438caa\") "
Nov 03 20:09:05.145044 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.145023    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.145531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.144358    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c91d62-295e-497e-a507-6244fa438caa-plugins-config-volume" pod "72c91d62-295e-497e-a507-6244fa438caa" (UID: "72c91d62-295e-497e-a507-6244fa438caa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:09:05.148577 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.148541    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-kube-api-access-xvghb" pod "f596e70f-a49c-47e8-8b8d-e5d27612f5c7" (UID: "f596e70f-a49c-47e8-8b8d-e5d27612f5c7"). InnerVolumeSpecName "kube-api-access-xvghb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:05.150817 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.150795    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c91d62-295e-497e-a507-6244fa438caa-kube-api-access-6g6jw" pod "72c91d62-295e-497e-a507-6244fa438caa" (UID: "72c91d62-295e-497e-a507-6244fa438caa"). InnerVolumeSpecName "kube-api-access-6g6jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:05.246599 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.246514    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvghb\" (UniqueName: \"kubernetes.io/projected/f596e70f-a49c-47e8-8b8d-e5d27612f5c7-kube-api-access-xvghb\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.246599 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.246554    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c91d62-295e-497e-a507-6244fa438caa-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.246599 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.246568    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g6jw\" (UniqueName: \"kubernetes.io/projected/72c91d62-295e-497e-a507-6244fa438caa-kube-api-access-6g6jw\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:05.555551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.555468    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"
Nov 03 20:09:05.555551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.555500    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg" event={"ID":"f596e70f-a49c-47e8-8b8d-e5d27612f5c7","Type":"ContainerDied","Data":"93d65541e3444a25ccceed090d6420cee0bddf3d39d2f03b9b4c457cf6f4b6f3"}
Nov 03 20:09:05.556079 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.555572    4531 scope.go:122] "RemoveContainer" containerID="52466bd853f4d7f9c59844944a312f5f0d6ecc0d2d13715d02ff1ce0cfb1fc16"
Nov 03 20:09:05.557012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.556986    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"
Nov 03 20:09:05.557143 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.557014    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj" event={"ID":"72c91d62-295e-497e-a507-6244fa438caa","Type":"ContainerDied","Data":"e41a554f6c9a9140a93578fb5ce89cc12f1ffa85573ca7502558ec9b73ab2c4f"}
Nov 03 20:09:05.558688 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.558643    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5" event={"ID":"f9542796-9bc6-4afc-b664-35a17397bc61","Type":"ContainerDied","Data":"c14e49a93a6aa4dc50a573de28220ac1726637b7f687f4d53c67185160024952"}
Nov 03 20:09:05.558820 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.558693    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"
Nov 03 20:09:05.567402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.567385    4531 scope.go:122] "RemoveContainer" containerID="02fd31d8e5afee285f86496f6e597632374cb4bdac1e4a6d27b43e1aaa511623"
Nov 03 20:09:05.577222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.577203    4531 scope.go:122] "RemoveContainer" containerID="d5c899d07753b75ea3fc9beab48520f756fa99113df7f362a333224a1297f2be"
Nov 03 20:09:05.585288 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.585256    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"]
Nov 03 20:09:05.587618 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.587595    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-jfbs5"]
Nov 03 20:09:05.595281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.595254    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"]
Nov 03 20:09:05.599109 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.599087    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0486e8-777477746c-44hdj"]
Nov 03 20:09:05.606723 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.606701    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"]
Nov 03 20:09:05.611004 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:05.610981    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0486e8-75b78f57d7-v2ndg"]
Nov 03 20:09:06.857741 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:06.857711    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c91d62-295e-497e-a507-6244fa438caa" path="/var/lib/kubelet/pods/72c91d62-295e-497e-a507-6244fa438caa/volumes"
Nov 03 20:09:06.858350 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:06.858335    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f596e70f-a49c-47e8-8b8d-e5d27612f5c7" path="/var/lib/kubelet/pods/f596e70f-a49c-47e8-8b8d-e5d27612f5c7/volumes"
Nov 03 20:09:06.858795 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:06.858780    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" path="/var/lib/kubelet/pods/f9542796-9bc6-4afc-b664-35a17397bc61/volumes"
Nov 03 20:09:07.584848 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:07.584801    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c547b4b-wcxhd" event={"ID":"cb11cb0b-8af0-4643-95eb-1318dcaf0d13","Type":"ContainerStarted","Data":"67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0"}
Nov 03 20:09:07.585038 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:07.584903    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:07.600904 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:07.600848    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-86c547b4b-wcxhd" podStartSLOduration=1.672076212 podStartE2EDuration="4.600834349s" totalImagesPullingTime="2.928758137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:07.599254804 +0000 UTC m=+2260.869378099" watchObservedRunningTime="2026-11-03 20:09:07.600834349 +0000 UTC m=+2260.870957645"
Nov 03 20:09:11.033424 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.033386    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:11.046295 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.046255    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"]
Nov 03 20:09:11.046959 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.046933    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy"
Nov 03 20:09:11.047094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.046961    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy"
Nov 03 20:09:11.047094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.046985    4531 cpu_manager.go:414] "containerMap: removing container" podUID="72c91d62-295e-497e-a507-6244fa438caa" containerName="payload-pre-processing"
Nov 03 20:09:11.047094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.046995    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c91d62-295e-497e-a507-6244fa438caa" containerName="payload-pre-processing"
Nov 03 20:09:11.047094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.047027    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f596e70f-a49c-47e8-8b8d-e5d27612f5c7" containerName="payload-processing"
Nov 03 20:09:11.047094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.047036    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f596e70f-a49c-47e8-8b8d-e5d27612f5c7" containerName="payload-processing"
Nov 03 20:09:11.047412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.047142    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c91d62-295e-497e-a507-6244fa438caa" containerName="payload-pre-processing"
Nov 03 20:09:11.047412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.047158    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f596e70f-a49c-47e8-8b8d-e5d27612f5c7" containerName="payload-processing"
Nov 03 20:09:11.047412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.047192    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9542796-9bc6-4afc-b664-35a17397bc61" containerName="istio-proxy"
Nov 03 20:09:11.051139 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.051118    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.063706 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.063673    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"]
Nov 03 20:09:11.106129 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.106091    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.106129 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.106133    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.106437 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.106162    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.106437 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.106330    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.106437 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.106393    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-888sc\" (UniqueName: \"kubernetes.io/projected/de3afce5-6597-4f19-8f0f-d4a695f4af39-kube-api-access-888sc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.106625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.106489    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.106625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.106541    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3afce5-6597-4f19-8f0f-d4a695f4af39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.106625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.106598    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.106804 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.106674    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.173965 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.173916    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:11.207456 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.207422    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.207666 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.207469    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.207666 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.207575    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.207666 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.207614    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.207666 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.207650    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.207899 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.207709    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.207899 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.207777    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-888sc\" (UniqueName: \"kubernetes.io/projected/de3afce5-6597-4f19-8f0f-d4a695f4af39-kube-api-access-888sc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.208025 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.207816    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.208094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.208026    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.208094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.208077    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.208246 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.208035    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3afce5-6597-4f19-8f0f-d4a695f4af39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.208368 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.208345    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.208464 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.208445    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.208718 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.208696    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3afce5-6597-4f19-8f0f-d4a695f4af39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.210448 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.210429    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.210652 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.210635    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.215320 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.215296    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3afce5-6597-4f19-8f0f-d4a695f4af39-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.215428 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.215373    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-888sc\" (UniqueName: \"kubernetes.io/projected/de3afce5-6597-4f19-8f0f-d4a695f4af39-kube-api-access-888sc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-tdppc\" (UID: \"de3afce5-6597-4f19-8f0f-d4a695f4af39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.305809 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.305714    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:11.368482 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.368442    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:11.557735 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.557648    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:11.649043 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.649020    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"]
Nov 03 20:09:11.651325 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:09:11.651152    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3afce5_6597_4f19_8f0f_d4a695f4af39.slice/crio-099aa5e1a03c033084fb51b139874726ffd4681d0dd3f4c45a867344473aae87 WatchSource:0}: Error finding container 099aa5e1a03c033084fb51b139874726ffd4681d0dd3f4c45a867344473aae87: Status 404 returned error can't find the container with id 099aa5e1a03c033084fb51b139874726ffd4681d0dd3f4c45a867344473aae87
Nov 03 20:09:11.653581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.653544    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:11.653674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.653618    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:11.653722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:11.653675    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:12.618220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:12.618165    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" event={"ID":"de3afce5-6597-4f19-8f0f-d4a695f4af39","Type":"ContainerStarted","Data":"4b5963dfab513b9045a6dec5a6da91d3986c684aa5828c99afcd5004a82a87a1"}
Nov 03 20:09:12.618662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:12.618237    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" event={"ID":"de3afce5-6597-4f19-8f0f-d4a695f4af39","Type":"ContainerStarted","Data":"099aa5e1a03c033084fb51b139874726ffd4681d0dd3f4c45a867344473aae87"}
Nov 03 20:09:12.636167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:12.636096    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" podStartSLOduration=1.636077167 podStartE2EDuration="1.636077167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:12.634534729 +0000 UTC m=+2265.904658024" watchObservedRunningTime="2026-11-03 20:09:12.636077167 +0000 UTC m=+2265.906200464"
Nov 03 20:09:13.369525 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:13.369486    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:13.371075 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:13.371046    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tdppc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 20:09:13.371261 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:13.371105    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" podUID="de3afce5-6597-4f19-8f0f-d4a695f4af39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 20:09:14.369676 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:14.369636    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tdppc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 20:09:14.370229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:14.369700    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" podUID="de3afce5-6597-4f19-8f0f-d4a695f4af39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 20:09:15.369824 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:15.369788    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tdppc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 20:09:15.370274 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:15.369844    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" podUID="de3afce5-6597-4f19-8f0f-d4a695f4af39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 20:09:16.369448 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:16.369402    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tdppc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 20:09:16.369659 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:16.369467    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" podUID="de3afce5-6597-4f19-8f0f-d4a695f4af39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 20:09:16.961854 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:16.961805    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:17.227504 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:17.227421    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:17.369081 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:17.369037    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tdppc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 20:09:17.369301 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:17.369096    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" podUID="de3afce5-6597-4f19-8f0f-d4a695f4af39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 20:09:18.369640 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:18.369606    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tdppc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 03 20:09:18.370071 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:18.369662    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" podUID="de3afce5-6597-4f19-8f0f-d4a695f4af39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 03 20:09:18.593614 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:18.593573    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:19.962840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:19.962803    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:19.963368 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:19.963224    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:19.964229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:19.964206    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc"
Nov 03 20:09:20.572682 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:09:20.572632    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:09:24.492781 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.492732    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"]
Nov 03 20:09:24.496484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.496463    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:24.500424 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.500392    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:09:24.500592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.500469    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:09:24.500592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.500500    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:09:24.500725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.500596    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:09:24.503029 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.503003    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"]
Nov 03 20:09:24.673212 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.673158    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbf9l\" (UniqueName: \"kubernetes.io/projected/87058489-0b28-480e-a0a7-80f1fdb72817-kube-api-access-mbf9l\") pod \"maas-api-revoke-sub-85237e45-2hgfz\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:24.673382 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.673288    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2hgfz\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:24.673382 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.673354    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2hgfz\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:24.774465 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.774370    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbf9l\" (UniqueName: \"kubernetes.io/projected/87058489-0b28-480e-a0a7-80f1fdb72817-kube-api-access-mbf9l\") pod \"maas-api-revoke-sub-85237e45-2hgfz\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:24.774665 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.774522    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2hgfz\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:24.774665 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.774543    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2hgfz\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:24.775192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.775153    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-2hgfz\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:24.776934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.776913    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-2hgfz\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:24.781351 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.781330    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbf9l\" (UniqueName: \"kubernetes.io/projected/87058489-0b28-480e-a0a7-80f1fdb72817-kube-api-access-mbf9l\") pod \"maas-api-revoke-sub-85237e45-2hgfz\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:24.815602 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:24.815568    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:25.018816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:25.018755    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"]
Nov 03 20:09:25.027204 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:25.023470    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67b44a7a-gthpm"]
Nov 03 20:09:25.091677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:25.091646    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"]
Nov 03 20:09:25.093055 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:09:25.093028    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87058489_0b28_480e_a0a7_80f1fdb72817.slice/crio-5a874278cdd6e7f2e675a7f6c35f41f20e1c566c39277a1fd0bf07f1a5fd987d WatchSource:0}: Error finding container 5a874278cdd6e7f2e675a7f6c35f41f20e1c566c39277a1fd0bf07f1a5fd987d: Status 404 returned error can't find the container with id 5a874278cdd6e7f2e675a7f6c35f41f20e1c566c39277a1fd0bf07f1a5fd987d
Nov 03 20:09:25.724479 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:25.724442    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz" event={"ID":"87058489-0b28-480e-a0a7-80f1fdb72817","Type":"ContainerStarted","Data":"6f980099b0729370ab11c0c3043f765bdac3b76697a8dbeff1a7dd718ad248ca"}
Nov 03 20:09:25.724908 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:25.724503    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz" event={"ID":"87058489-0b28-480e-a0a7-80f1fdb72817","Type":"ContainerStarted","Data":"5a874278cdd6e7f2e675a7f6c35f41f20e1c566c39277a1fd0bf07f1a5fd987d"}
Nov 03 20:09:25.740467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:25.740406    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz" podStartSLOduration=1.740389942 podStartE2EDuration="1.740389942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:25.736997241 +0000 UTC m=+2279.007120536" watchObservedRunningTime="2026-11-03 20:09:25.740389942 +0000 UTC m=+2279.010513237"
Nov 03 20:09:26.733568 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:26.733527    4531 generic.go:356] "Generic (PLEG): container finished" podID="87058489-0b28-480e-a0a7-80f1fdb72817" containerID="6f980099b0729370ab11c0c3043f765bdac3b76697a8dbeff1a7dd718ad248ca" exitCode=0
Nov 03 20:09:26.734023 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:26.733608    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz" event={"ID":"87058489-0b28-480e-a0a7-80f1fdb72817","Type":"ContainerDied","Data":"6f980099b0729370ab11c0c3043f765bdac3b76697a8dbeff1a7dd718ad248ca"}
Nov 03 20:09:26.857895 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:26.857862    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00be0ef0-9ee6-4f14-a738-523b34ebb8bd" path="/var/lib/kubelet/pods/00be0ef0-9ee6-4f14-a738-523b34ebb8bd/volumes"
Nov 03 20:09:27.894210 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:27.894168    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:28.011598 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.011516    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87058489-0b28-480e-a0a7-80f1fdb72817-kube-api-access-mbf9l\" (UniqueName: \"kubernetes.io/projected/87058489-0b28-480e-a0a7-80f1fdb72817-kube-api-access-mbf9l\") pod \"87058489-0b28-480e-a0a7-80f1fdb72817\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") "
Nov 03 20:09:28.011767 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.011613    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-xks-ca\") pod \"87058489-0b28-480e-a0a7-80f1fdb72817\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") "
Nov 03 20:09:28.011767 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.011668    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-openshift-ca\") pod \"87058489-0b28-480e-a0a7-80f1fdb72817\" (UID: \"87058489-0b28-480e-a0a7-80f1fdb72817\") "
Nov 03 20:09:28.012033 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.012009    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-openshift-ca" pod "87058489-0b28-480e-a0a7-80f1fdb72817" (UID: "87058489-0b28-480e-a0a7-80f1fdb72817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:09:28.013694 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.013668    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87058489-0b28-480e-a0a7-80f1fdb72817-kube-api-access-mbf9l" pod "87058489-0b28-480e-a0a7-80f1fdb72817" (UID: "87058489-0b28-480e-a0a7-80f1fdb72817"). InnerVolumeSpecName "kube-api-access-mbf9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:28.013830 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.013808    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-xks-ca" pod "87058489-0b28-480e-a0a7-80f1fdb72817" (UID: "87058489-0b28-480e-a0a7-80f1fdb72817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:09:28.113010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.112966    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbf9l\" (UniqueName: \"kubernetes.io/projected/87058489-0b28-480e-a0a7-80f1fdb72817-kube-api-access-mbf9l\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:28.113010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.113006    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:28.113010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.113016    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87058489-0b28-480e-a0a7-80f1fdb72817-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:28.748124 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.748095    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"
Nov 03 20:09:28.748341 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.748094    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz" event={"ID":"87058489-0b28-480e-a0a7-80f1fdb72817","Type":"ContainerDied","Data":"5a874278cdd6e7f2e675a7f6c35f41f20e1c566c39277a1fd0bf07f1a5fd987d"}
Nov 03 20:09:28.748341 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:28.748244    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a874278cdd6e7f2e675a7f6c35f41f20e1c566c39277a1fd0bf07f1a5fd987d"
Nov 03 20:09:35.759874 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:35.759841    4531 scope.go:122] "RemoveContainer" containerID="c96f36b6e713c2e0a1d48a8450037894d8e0fd97e35354d8676f0d6c1e4da0a4"
Nov 03 20:09:35.770344 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:35.770324    4531 scope.go:122] "RemoveContainer" containerID="d45f142b74deb59662e0aae05ee90558c54f0d711eaa4b1e64c0163be11084b2"
Nov 03 20:09:35.779232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:35.779203    4531 scope.go:122] "RemoveContainer" containerID="ca32d590779be2edab6a6e74c4e9175119dd5cd9149eaffa86a449aed8c2bec1"
Nov 03 20:09:35.986719 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:35.986681    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:36.080784 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.080694    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86c547b4b-wcxhd"]
Nov 03 20:09:36.080956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.080899    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-86c547b4b-wcxhd" podUID="cb11cb0b-8af0-4643-95eb-1318dcaf0d13" containerName="manager" containerID="cri-o://67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0" gracePeriod=30
Nov 03 20:09:36.334269 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.334212    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:36.391185 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.391147    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-webhook-cert\") pod \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") "
Nov 03 20:09:36.391384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.391273    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-kube-api-access-dnlz8\" (UniqueName: \"kubernetes.io/projected/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-kube-api-access-dnlz8\") pod \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") "
Nov 03 20:09:36.391384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.391331    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-metrics-certs\") pod \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\" (UID: \"cb11cb0b-8af0-4643-95eb-1318dcaf0d13\") "
Nov 03 20:09:36.393519 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.393409    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-webhook-cert" pod "cb11cb0b-8af0-4643-95eb-1318dcaf0d13" (UID: "cb11cb0b-8af0-4643-95eb-1318dcaf0d13"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:09:36.393519 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.393419    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-kube-api-access-dnlz8" pod "cb11cb0b-8af0-4643-95eb-1318dcaf0d13" (UID: "cb11cb0b-8af0-4643-95eb-1318dcaf0d13"). InnerVolumeSpecName "kube-api-access-dnlz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:36.393727 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.393583    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-metrics-certs" pod "cb11cb0b-8af0-4643-95eb-1318dcaf0d13" (UID: "cb11cb0b-8af0-4643-95eb-1318dcaf0d13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:09:36.492453 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.492411    4531 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-webhook-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:36.492453 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.492448    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnlz8\" (UniqueName: \"kubernetes.io/projected/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-kube-api-access-dnlz8\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:36.492453 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.492459    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb11cb0b-8af0-4643-95eb-1318dcaf0d13-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:36.812264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.812225    4531 generic.go:356] "Generic (PLEG): container finished" podID="cb11cb0b-8af0-4643-95eb-1318dcaf0d13" containerID="67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0" exitCode=0
Nov 03 20:09:36.812699 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.812281    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c547b4b-wcxhd"
Nov 03 20:09:36.812699 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.812319    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c547b4b-wcxhd" event={"ID":"cb11cb0b-8af0-4643-95eb-1318dcaf0d13","Type":"ContainerDied","Data":"67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0"}
Nov 03 20:09:36.812699 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.812386    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c547b4b-wcxhd" event={"ID":"cb11cb0b-8af0-4643-95eb-1318dcaf0d13","Type":"ContainerDied","Data":"a1d0803c7ce5cedbcd83b6270ea0c82c63fe65323b27783d9979e32c30741430"}
Nov 03 20:09:36.812699 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.812422    4531 scope.go:122] "RemoveContainer" containerID="67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0"
Nov 03 20:09:36.822369 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.822347    4531 scope.go:122] "RemoveContainer" containerID="67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0"
Nov 03 20:09:36.822695 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:09:36.822674    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0\": container with ID starting with 67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0 not found: ID does not exist" containerID="67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0"
Nov 03 20:09:36.822745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.822707    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0"} err="failed to get container status \"67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0\": rpc error: code = NotFound desc = could not find container \"67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0\": container with ID starting with 67cab443e5e90636b36e971235add23e6b4ab6a1d91643e8375b8e0373fa18b0 not found: ID does not exist"
Nov 03 20:09:36.832659 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.832629    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86c547b4b-wcxhd"]
Nov 03 20:09:36.838291 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.838264    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86c547b4b-wcxhd"]
Nov 03 20:09:36.864608 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:36.864575    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb11cb0b-8af0-4643-95eb-1318dcaf0d13" path="/var/lib/kubelet/pods/cb11cb0b-8af0-4643-95eb-1318dcaf0d13/volumes"
Nov 03 20:09:37.860624 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.860588    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:37.958475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.958432    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86c547b4b-x4k2g"]
Nov 03 20:09:37.958872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.958856    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cb11cb0b-8af0-4643-95eb-1318dcaf0d13" containerName="manager"
Nov 03 20:09:37.958928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.958872    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb11cb0b-8af0-4643-95eb-1318dcaf0d13" containerName="manager"
Nov 03 20:09:37.958928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.958898    4531 cpu_manager.go:414] "containerMap: removing container" podUID="87058489-0b28-480e-a0a7-80f1fdb72817" containerName="revoke-keys"
Nov 03 20:09:37.958928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.958903    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87058489-0b28-480e-a0a7-80f1fdb72817" containerName="revoke-keys"
Nov 03 20:09:37.959046 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.958977    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="87058489-0b28-480e-a0a7-80f1fdb72817" containerName="revoke-keys"
Nov 03 20:09:37.959046 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.958989    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb11cb0b-8af0-4643-95eb-1318dcaf0d13" containerName="manager"
Nov 03 20:09:37.963786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.963759    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:37.969522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.969496    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 20:09:37.969850 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.969824    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5cw4d\""
Nov 03 20:09:37.969956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.969862    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 20:09:37.969956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.969923    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 20:09:37.970695 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:37.970667    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86c547b4b-x4k2g"]
Nov 03 20:09:38.008960 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.008929    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct45l\" (UniqueName: \"kubernetes.io/projected/9bd3bb4d-1444-4ce0-b926-111f7bc54813-kube-api-access-ct45l\") pod \"maas-controller-86c547b4b-x4k2g\" (UID: \"9bd3bb4d-1444-4ce0-b926-111f7bc54813\") " pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:38.009141 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.008975    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bd3bb4d-1444-4ce0-b926-111f7bc54813-webhook-cert\") pod \"maas-controller-86c547b4b-x4k2g\" (UID: \"9bd3bb4d-1444-4ce0-b926-111f7bc54813\") " pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:38.009141 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.009123    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bd3bb4d-1444-4ce0-b926-111f7bc54813-metrics-certs\") pod \"maas-controller-86c547b4b-x4k2g\" (UID: \"9bd3bb4d-1444-4ce0-b926-111f7bc54813\") " pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:38.110319 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.110281    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct45l\" (UniqueName: \"kubernetes.io/projected/9bd3bb4d-1444-4ce0-b926-111f7bc54813-kube-api-access-ct45l\") pod \"maas-controller-86c547b4b-x4k2g\" (UID: \"9bd3bb4d-1444-4ce0-b926-111f7bc54813\") " pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:38.110528 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.110331    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bd3bb4d-1444-4ce0-b926-111f7bc54813-webhook-cert\") pod \"maas-controller-86c547b4b-x4k2g\" (UID: \"9bd3bb4d-1444-4ce0-b926-111f7bc54813\") " pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:38.110528 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.110410    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bd3bb4d-1444-4ce0-b926-111f7bc54813-metrics-certs\") pod \"maas-controller-86c547b4b-x4k2g\" (UID: \"9bd3bb4d-1444-4ce0-b926-111f7bc54813\") " pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:38.112997 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.112930    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bd3bb4d-1444-4ce0-b926-111f7bc54813-metrics-certs\") pod \"maas-controller-86c547b4b-x4k2g\" (UID: \"9bd3bb4d-1444-4ce0-b926-111f7bc54813\") " pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:38.112997 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.112950    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9bd3bb4d-1444-4ce0-b926-111f7bc54813-webhook-cert\") pod \"maas-controller-86c547b4b-x4k2g\" (UID: \"9bd3bb4d-1444-4ce0-b926-111f7bc54813\") " pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:38.117351 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.117327    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct45l\" (UniqueName: \"kubernetes.io/projected/9bd3bb4d-1444-4ce0-b926-111f7bc54813-kube-api-access-ct45l\") pod \"maas-controller-86c547b4b-x4k2g\" (UID: \"9bd3bb4d-1444-4ce0-b926-111f7bc54813\") " pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:38.279233 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.279201    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:38.559151 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.559117    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86c547b4b-x4k2g"]
Nov 03 20:09:38.560774 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:09:38.560745    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd3bb4d_1444_4ce0_b926_111f7bc54813.slice/crio-fee750056bc50188829d0583eb01877906b596a9ea20d933e1d463a2f844ad1a WatchSource:0}: Error finding container fee750056bc50188829d0583eb01877906b596a9ea20d933e1d463a2f844ad1a: Status 404 returned error can't find the container with id fee750056bc50188829d0583eb01877906b596a9ea20d933e1d463a2f844ad1a
Nov 03 20:09:38.826768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:38.826677    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c547b4b-x4k2g" event={"ID":"9bd3bb4d-1444-4ce0-b926-111f7bc54813","Type":"ContainerStarted","Data":"fee750056bc50188829d0583eb01877906b596a9ea20d933e1d463a2f844ad1a"}
Nov 03 20:09:39.836215 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:39.836106    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c547b4b-x4k2g" event={"ID":"9bd3bb4d-1444-4ce0-b926-111f7bc54813","Type":"ContainerStarted","Data":"b1612e52be2c20870de8fe66b19db0ad6c631c1d07768a50985e1fac1d5524fc"}
Nov 03 20:09:39.836632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:39.836241    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:09:39.853045 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:39.852986    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-86c547b4b-x4k2g" podStartSLOduration=2.451179295 podStartE2EDuration="2.852971586s" totalImagesPullingTime="401.792291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:39.849737201 +0000 UTC m=+2293.119860499" watchObservedRunningTime="2026-11-03 20:09:39.852971586 +0000 UTC m=+2293.123094881"
Nov 03 20:09:50.573238 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:09:50.573194    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:09:50.844818 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:09:50.844729    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86c547b4b-x4k2g"
Nov 03 20:10:17.103391 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.103348    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"]
Nov 03 20:10:17.107437 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.107414    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:17.110516 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.110487    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:10:17.111633 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.110524    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:10:17.111633 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.110589    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:10:17.111877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.110807    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:10:17.113821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.113798    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"]
Nov 03 20:10:17.152993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.152954    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjvs\" (UniqueName: \"kubernetes.io/projected/df1f3cbd-31d0-4191-a720-33af8cb17560-kube-api-access-rvjvs\") pod \"maas-api-revoke-sub-c1e271a9-p9bgh\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:17.153213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.153027    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-p9bgh\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:17.153213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.153095    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-p9bgh\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:17.254135 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.254089    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-p9bgh\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:17.254135 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.254139    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-p9bgh\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:17.254392 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.254288    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjvs\" (UniqueName: \"kubernetes.io/projected/df1f3cbd-31d0-4191-a720-33af8cb17560-kube-api-access-rvjvs\") pod \"maas-api-revoke-sub-c1e271a9-p9bgh\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:17.254915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.254879    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-p9bgh\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:17.256620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.256595    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-p9bgh\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:17.261564 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.261538    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjvs\" (UniqueName: \"kubernetes.io/projected/df1f3cbd-31d0-4191-a720-33af8cb17560-kube-api-access-rvjvs\") pod \"maas-api-revoke-sub-c1e271a9-p9bgh\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:17.295934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.295897    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:17.424249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.424212    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:17.696969 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:17.696935    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"]
Nov 03 20:10:17.698341 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:10:17.698308    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1f3cbd_31d0_4191_a720_33af8cb17560.slice/crio-cf30b1c4fd0d25f333f2d7e410850fd0c893cfd180e8070055ddd36c409fca97 WatchSource:0}: Error finding container cf30b1c4fd0d25f333f2d7e410850fd0c893cfd180e8070055ddd36c409fca97: Status 404 returned error can't find the container with id cf30b1c4fd0d25f333f2d7e410850fd0c893cfd180e8070055ddd36c409fca97
Nov 03 20:10:18.118965 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:18.118926    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh" event={"ID":"df1f3cbd-31d0-4191-a720-33af8cb17560","Type":"ContainerStarted","Data":"3a97ad7c7dcf7a681cb04f07d77592729295cac894f69e1065047fd4c424f795"}
Nov 03 20:10:18.119395 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:18.118985    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh" event={"ID":"df1f3cbd-31d0-4191-a720-33af8cb17560","Type":"ContainerStarted","Data":"cf30b1c4fd0d25f333f2d7e410850fd0c893cfd180e8070055ddd36c409fca97"}
Nov 03 20:10:18.134265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:18.134211    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh" podStartSLOduration=1.13419683 podStartE2EDuration="1.13419683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:18.132141547 +0000 UTC m=+2331.402264843" watchObservedRunningTime="2026-11-03 20:10:18.13419683 +0000 UTC m=+2331.404320124"
Nov 03 20:10:18.836915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:18.836824    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:18.970644 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:18.970605    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:19.126803 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:19.126698    4531 generic.go:356] "Generic (PLEG): container finished" podID="df1f3cbd-31d0-4191-a720-33af8cb17560" containerID="3a97ad7c7dcf7a681cb04f07d77592729295cac894f69e1065047fd4c424f795" exitCode=0
Nov 03 20:10:19.126803 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:19.126787    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh" event={"ID":"df1f3cbd-31d0-4191-a720-33af8cb17560","Type":"ContainerDied","Data":"3a97ad7c7dcf7a681cb04f07d77592729295cac894f69e1065047fd4c424f795"}
Nov 03 20:10:20.264808 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:20.264781    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:20.386628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:20.386536    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-openshift-ca\") pod \"df1f3cbd-31d0-4191-a720-33af8cb17560\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") "
Nov 03 20:10:20.386628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:20.386589    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-xks-ca\") pod \"df1f3cbd-31d0-4191-a720-33af8cb17560\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") "
Nov 03 20:10:20.386880 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:20.386655    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1f3cbd-31d0-4191-a720-33af8cb17560-kube-api-access-rvjvs\" (UniqueName: \"kubernetes.io/projected/df1f3cbd-31d0-4191-a720-33af8cb17560-kube-api-access-rvjvs\") pod \"df1f3cbd-31d0-4191-a720-33af8cb17560\" (UID: \"df1f3cbd-31d0-4191-a720-33af8cb17560\") "
Nov 03 20:10:20.386955 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:20.386912    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-openshift-ca" pod "df1f3cbd-31d0-4191-a720-33af8cb17560" (UID: "df1f3cbd-31d0-4191-a720-33af8cb17560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:10:20.387037 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:20.387021    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:20.388812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:20.388783    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-xks-ca" pod "df1f3cbd-31d0-4191-a720-33af8cb17560" (UID: "df1f3cbd-31d0-4191-a720-33af8cb17560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:10:20.388812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:20.388798    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1f3cbd-31d0-4191-a720-33af8cb17560-kube-api-access-rvjvs" pod "df1f3cbd-31d0-4191-a720-33af8cb17560" (UID: "df1f3cbd-31d0-4191-a720-33af8cb17560"). InnerVolumeSpecName "kube-api-access-rvjvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:20.488429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:20.488390    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1f3cbd-31d0-4191-a720-33af8cb17560-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:20.488429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:20.488423    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjvs\" (UniqueName: \"kubernetes.io/projected/df1f3cbd-31d0-4191-a720-33af8cb17560-kube-api-access-rvjvs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:20.573745 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:10:20.573708    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:10:21.139984 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:21.139949    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"
Nov 03 20:10:21.140146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:21.139979    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh" event={"ID":"df1f3cbd-31d0-4191-a720-33af8cb17560","Type":"ContainerDied","Data":"cf30b1c4fd0d25f333f2d7e410850fd0c893cfd180e8070055ddd36c409fca97"}
Nov 03 20:10:21.140146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:21.140029    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf30b1c4fd0d25f333f2d7e410850fd0c893cfd180e8070055ddd36c409fca97"
Nov 03 20:10:33.019081 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:33.019041    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"]
Nov 03 20:10:33.023601 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:33.023572    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442859bb-4nnxv"]
Nov 03 20:10:34.856698 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:34.856664    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c3e245-8841-403a-a185-1de590d57058" path="/var/lib/kubelet/pods/b0c3e245-8841-403a-a185-1de590d57058/volumes"
Nov 03 20:10:35.860715 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.860680    4531 scope.go:122] "RemoveContainer" containerID="7f3629ba7b7d3ccec38f2f4dfca22feb207b4555ee9aa240ee5bec79533d8945"
Nov 03 20:10:35.870479 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.870457    4531 scope.go:122] "RemoveContainer" containerID="091436b32a26221aae57bc73ad4173b0422f3345ede26daa45b776df03102dce"
Nov 03 20:10:35.879814 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.879789    4531 scope.go:122] "RemoveContainer" containerID="2b427e29beb241d47623a53361a9f4ada9f2e7049a30ccd3c9c8073ce03c4df9"
Nov 03 20:10:35.888956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.888928    4531 scope.go:122] "RemoveContainer" containerID="665d2783255fca8e1c89527b8f8b3d98f099743bade2e132df46894f964ab330"
Nov 03 20:10:35.900650 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.900619    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"]
Nov 03 20:10:35.901053 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.901039    4531 cpu_manager.go:414] "containerMap: removing container" podUID="df1f3cbd-31d0-4191-a720-33af8cb17560" containerName="revoke-keys"
Nov 03 20:10:35.901111 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.901055    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1f3cbd-31d0-4191-a720-33af8cb17560" containerName="revoke-keys"
Nov 03 20:10:35.901146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.901137    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1f3cbd-31d0-4191-a720-33af8cb17560" containerName="revoke-keys"
Nov 03 20:10:35.909477 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.909449    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:35.912165 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.912093    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:10:35.912414 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.912392    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:10:35.913050 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.913025    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"]
Nov 03 20:10:35.913319 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.913297    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:10:35.913723 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:35.913684    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:10:36.099661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:36.099625    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rl2nr\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:36.099846 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:36.099679    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rl2nr\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:36.099846 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:36.099805    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g62v4\" (UniqueName: \"kubernetes.io/projected/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-kube-api-access-g62v4\") pod \"maas-api-revoke-sub-2e1f5b5f-rl2nr\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:36.200472 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:36.200429    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rl2nr\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:36.200673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:36.200512    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g62v4\" (UniqueName: \"kubernetes.io/projected/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-kube-api-access-g62v4\") pod \"maas-api-revoke-sub-2e1f5b5f-rl2nr\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:36.200673 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:36.200572    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rl2nr\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:36.201230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:36.201206    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rl2nr\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:36.202980 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:36.202958    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rl2nr\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:36.207774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:36.207748    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g62v4\" (UniqueName: \"kubernetes.io/projected/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-kube-api-access-g62v4\") pod \"maas-api-revoke-sub-2e1f5b5f-rl2nr\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:36.227411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:36.227383    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:36.502165 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:36.502137    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"]
Nov 03 20:10:36.504168 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:10:36.504140    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee744b4_c817_42d0_8b8d_74cb9da5c4de.slice/crio-68d74c915487f10f8a7207ca732513566460f6906ba4ca1a4a89ed1dcdc7da62 WatchSource:0}: Error finding container 68d74c915487f10f8a7207ca732513566460f6906ba4ca1a4a89ed1dcdc7da62: Status 404 returned error can't find the container with id 68d74c915487f10f8a7207ca732513566460f6906ba4ca1a4a89ed1dcdc7da62
Nov 03 20:10:37.262486 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:37.262449    4531 generic.go:356] "Generic (PLEG): container finished" podID="3ee744b4-c817-42d0-8b8d-74cb9da5c4de" containerID="46cbb80ec590131a87cb0df90ab5ffbb24796566e9ab02779a79f96605fbfff3" exitCode=0
Nov 03 20:10:37.262942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:37.262538    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr" event={"ID":"3ee744b4-c817-42d0-8b8d-74cb9da5c4de","Type":"ContainerDied","Data":"46cbb80ec590131a87cb0df90ab5ffbb24796566e9ab02779a79f96605fbfff3"}
Nov 03 20:10:37.262942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:37.262607    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr" event={"ID":"3ee744b4-c817-42d0-8b8d-74cb9da5c4de","Type":"ContainerStarted","Data":"68d74c915487f10f8a7207ca732513566460f6906ba4ca1a4a89ed1dcdc7da62"}
Nov 03 20:10:38.414071 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:38.414033    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:38.421495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:38.421465    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-xks-ca\") pod \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") "
Nov 03 20:10:38.421635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:38.421505    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-openshift-ca\") pod \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") "
Nov 03 20:10:38.421635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:38.421619    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-kube-api-access-g62v4\" (UniqueName: \"kubernetes.io/projected/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-kube-api-access-g62v4\") pod \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\" (UID: \"3ee744b4-c817-42d0-8b8d-74cb9da5c4de\") "
Nov 03 20:10:38.421905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:38.421883    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-openshift-ca" pod "3ee744b4-c817-42d0-8b8d-74cb9da5c4de" (UID: "3ee744b4-c817-42d0-8b8d-74cb9da5c4de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:10:38.423750 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:38.423723    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-kube-api-access-g62v4" pod "3ee744b4-c817-42d0-8b8d-74cb9da5c4de" (UID: "3ee744b4-c817-42d0-8b8d-74cb9da5c4de"). InnerVolumeSpecName "kube-api-access-g62v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:38.423837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:38.423787    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-xks-ca" pod "3ee744b4-c817-42d0-8b8d-74cb9da5c4de" (UID: "3ee744b4-c817-42d0-8b8d-74cb9da5c4de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:10:38.522914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:38.522821    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g62v4\" (UniqueName: \"kubernetes.io/projected/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-kube-api-access-g62v4\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:38.522914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:38.522864    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:38.522914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:38.522879    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ee744b4-c817-42d0-8b8d-74cb9da5c4de-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:39.015668 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:39.015631    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:39.276150 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:39.276120    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"
Nov 03 20:10:39.276150 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:39.276144    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr" event={"ID":"3ee744b4-c817-42d0-8b8d-74cb9da5c4de","Type":"ContainerDied","Data":"68d74c915487f10f8a7207ca732513566460f6906ba4ca1a4a89ed1dcdc7da62"}
Nov 03 20:10:39.276395 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:39.276212    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68d74c915487f10f8a7207ca732513566460f6906ba4ca1a4a89ed1dcdc7da62"
Nov 03 20:10:46.829538 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.829498    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"]
Nov 03 20:10:46.832222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.829941    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3ee744b4-c817-42d0-8b8d-74cb9da5c4de" containerName="revoke-keys"
Nov 03 20:10:46.832222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.829952    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee744b4-c817-42d0-8b8d-74cb9da5c4de" containerName="revoke-keys"
Nov 03 20:10:46.832222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.830023    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee744b4-c817-42d0-8b8d-74cb9da5c4de" containerName="revoke-keys"
Nov 03 20:10:46.833144 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.833121    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:46.839146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.836980    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:10:46.839146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.837025    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:10:46.839146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.837024    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:10:46.839146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.837130    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:10:46.839146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.838353    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"]
Nov 03 20:10:46.910412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.910369    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv4pr\" (UniqueName: \"kubernetes.io/projected/f761666e-87a9-4ff6-9309-490667dcd1c8-kube-api-access-lv4pr\") pod \"maas-api-revoke-sub-ec843018-hwr7w\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:46.910621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.910463    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hwr7w\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:46.910621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:46.910522    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hwr7w\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:47.011297 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:47.011256    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv4pr\" (UniqueName: \"kubernetes.io/projected/f761666e-87a9-4ff6-9309-490667dcd1c8-kube-api-access-lv4pr\") pod \"maas-api-revoke-sub-ec843018-hwr7w\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:47.011297 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:47.011298    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hwr7w\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:47.011607 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:47.011354    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hwr7w\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:47.012225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:47.012188    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hwr7w\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:47.014025 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:47.013997    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hwr7w\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:47.019094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:47.019067    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv4pr\" (UniqueName: \"kubernetes.io/projected/f761666e-87a9-4ff6-9309-490667dcd1c8-kube-api-access-lv4pr\") pod \"maas-api-revoke-sub-ec843018-hwr7w\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:47.149545 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:47.149507    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:47.442961 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:47.442935    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"]
Nov 03 20:10:47.445133 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:10:47.445099    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf761666e_87a9_4ff6_9309_490667dcd1c8.slice/crio-f46c3619cdece23a163e08d0fe531ce8fd86d2948594dc46b1931d059cded203 WatchSource:0}: Error finding container f46c3619cdece23a163e08d0fe531ce8fd86d2948594dc46b1931d059cded203: Status 404 returned error can't find the container with id f46c3619cdece23a163e08d0fe531ce8fd86d2948594dc46b1931d059cded203
Nov 03 20:10:48.331750 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:48.331711    4531 generic.go:356] "Generic (PLEG): container finished" podID="f761666e-87a9-4ff6-9309-490667dcd1c8" containerID="013bd4c59f99dd49e5dd5c304e0295e7acf6ef7354fc1b2b6911d07682b2e0ac" exitCode=0
Nov 03 20:10:48.332260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:48.331802    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w" event={"ID":"f761666e-87a9-4ff6-9309-490667dcd1c8","Type":"ContainerDied","Data":"013bd4c59f99dd49e5dd5c304e0295e7acf6ef7354fc1b2b6911d07682b2e0ac"}
Nov 03 20:10:48.332260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:48.331876    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w" event={"ID":"f761666e-87a9-4ff6-9309-490667dcd1c8","Type":"ContainerStarted","Data":"f46c3619cdece23a163e08d0fe531ce8fd86d2948594dc46b1931d059cded203"}
Nov 03 20:10:49.532529 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:49.532492    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:49.640546 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:49.640486    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-openshift-ca\") pod \"f761666e-87a9-4ff6-9309-490667dcd1c8\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") "
Nov 03 20:10:49.640546 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:49.640550    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f761666e-87a9-4ff6-9309-490667dcd1c8-kube-api-access-lv4pr\" (UniqueName: \"kubernetes.io/projected/f761666e-87a9-4ff6-9309-490667dcd1c8-kube-api-access-lv4pr\") pod \"f761666e-87a9-4ff6-9309-490667dcd1c8\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") "
Nov 03 20:10:49.640867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:49.640706    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-xks-ca\") pod \"f761666e-87a9-4ff6-9309-490667dcd1c8\" (UID: \"f761666e-87a9-4ff6-9309-490667dcd1c8\") "
Nov 03 20:10:49.641017 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:49.640987    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-openshift-ca" pod "f761666e-87a9-4ff6-9309-490667dcd1c8" (UID: "f761666e-87a9-4ff6-9309-490667dcd1c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:10:49.641151 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:49.641130    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:49.643627 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:49.643478    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f761666e-87a9-4ff6-9309-490667dcd1c8-kube-api-access-lv4pr" pod "f761666e-87a9-4ff6-9309-490667dcd1c8" (UID: "f761666e-87a9-4ff6-9309-490667dcd1c8"). InnerVolumeSpecName "kube-api-access-lv4pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:49.643627 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:49.643566    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-xks-ca" pod "f761666e-87a9-4ff6-9309-490667dcd1c8" (UID: "f761666e-87a9-4ff6-9309-490667dcd1c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:10:49.742643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:49.742595    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f761666e-87a9-4ff6-9309-490667dcd1c8-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:49.742643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:49.742642    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv4pr\" (UniqueName: \"kubernetes.io/projected/f761666e-87a9-4ff6-9309-490667dcd1c8-kube-api-access-lv4pr\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:50.383198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:50.375909    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w" event={"ID":"f761666e-87a9-4ff6-9309-490667dcd1c8","Type":"ContainerDied","Data":"f46c3619cdece23a163e08d0fe531ce8fd86d2948594dc46b1931d059cded203"}
Nov 03 20:10:50.383198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:50.375983    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46c3619cdece23a163e08d0fe531ce8fd86d2948594dc46b1931d059cded203"
Nov 03 20:10:50.383198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:50.376065    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"
Nov 03 20:10:50.737268 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:10:50.736794    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:10:50.803199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:50.802354    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"]
Nov 03 20:10:50.803199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:50.802615    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv" podUID="67155b07-975c-4a86-9ef2-39587de0fc2e" containerName="discovery" containerID="cri-o://19d92f501dbead2552582e16f4b69b6d164e3309d108220fc70d0cc1fda40fa0" gracePeriod=30
Nov 03 20:10:51.030613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.030261    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tdppc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:10:51.030613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.030328    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" podUID="de3afce5-6597-4f19-8f0f-d4a695f4af39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:10:51.805937 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.805893    4531 generic.go:356] "Generic (PLEG): container finished" podID="67155b07-975c-4a86-9ef2-39587de0fc2e" containerID="19d92f501dbead2552582e16f4b69b6d164e3309d108220fc70d0cc1fda40fa0" exitCode=0
Nov 03 20:10:51.810949 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.810912    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv" event={"ID":"67155b07-975c-4a86-9ef2-39587de0fc2e","Type":"ContainerDied","Data":"19d92f501dbead2552582e16f4b69b6d164e3309d108220fc70d0cc1fda40fa0"}
Nov 03 20:10:51.824189 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.824054    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:10:51.877550 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.877483    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-t5g9v" podUID="c6beddf2-5be3-42ae-a6c7-6256834bd41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:10:51.896208 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.896046    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 20:10:51.978409 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.978354    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-dns-cert\") pod \"67155b07-975c-4a86-9ef2-39587de0fc2e\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") "
Nov 03 20:10:51.978617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.978429    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67155b07-975c-4a86-9ef2-39587de0fc2e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67155b07-975c-4a86-9ef2-39587de0fc2e-local-certs\") pod \"67155b07-975c-4a86-9ef2-39587de0fc2e\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") "
Nov 03 20:10:51.978617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.978469    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-ca-configmap\") pod \"67155b07-975c-4a86-9ef2-39587de0fc2e\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") "
Nov 03 20:10:51.978617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.978524    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-token\" (UniqueName: \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-token\") pod \"67155b07-975c-4a86-9ef2-39587de0fc2e\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") "
Nov 03 20:10:51.978617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.978561    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-cacerts\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-cacerts\") pod \"67155b07-975c-4a86-9ef2-39587de0fc2e\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") "
Nov 03 20:10:51.978851 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.978621    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-kubeconfig\") pod \"67155b07-975c-4a86-9ef2-39587de0fc2e\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") "
Nov 03 20:10:51.978851 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.978651    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-kube-api-access-v2vnp\" (UniqueName: \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-kube-api-access-v2vnp\") pod \"67155b07-975c-4a86-9ef2-39587de0fc2e\" (UID: \"67155b07-975c-4a86-9ef2-39587de0fc2e\") "
Nov 03 20:10:51.979665 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.979608    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-ca-configmap" pod "67155b07-975c-4a86-9ef2-39587de0fc2e" (UID: "67155b07-975c-4a86-9ef2-39587de0fc2e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:10:51.982498 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.982465    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67155b07-975c-4a86-9ef2-39587de0fc2e-local-certs" pod "67155b07-975c-4a86-9ef2-39587de0fc2e" (UID: "67155b07-975c-4a86-9ef2-39587de0fc2e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:51.983082 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.983027    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-dns-cert" pod "67155b07-975c-4a86-9ef2-39587de0fc2e" (UID: "67155b07-975c-4a86-9ef2-39587de0fc2e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:10:51.983771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.983717    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-kubeconfig" pod "67155b07-975c-4a86-9ef2-39587de0fc2e" (UID: "67155b07-975c-4a86-9ef2-39587de0fc2e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:10:51.983967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.983938    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-token" pod "67155b07-975c-4a86-9ef2-39587de0fc2e" (UID: "67155b07-975c-4a86-9ef2-39587de0fc2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:51.984214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.984147    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-kube-api-access-v2vnp" pod "67155b07-975c-4a86-9ef2-39587de0fc2e" (UID: "67155b07-975c-4a86-9ef2-39587de0fc2e"). InnerVolumeSpecName "kube-api-access-v2vnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:51.984753 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:51.984723    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-cacerts" pod "67155b07-975c-4a86-9ef2-39587de0fc2e" (UID: "67155b07-975c-4a86-9ef2-39587de0fc2e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:10:52.080609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.080518    4531 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-dns-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:52.080609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.080551    4531 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/67155b07-975c-4a86-9ef2-39587de0fc2e-local-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:52.080609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.080561    4531 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-csr-ca-configmap\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:52.080609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.080576    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:52.080609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.080589    4531 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-cacerts\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:52.080609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.080599    4531 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67155b07-975c-4a86-9ef2-39587de0fc2e-istio-kubeconfig\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:52.080609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.080607    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2vnp\" (UniqueName: \"kubernetes.io/projected/67155b07-975c-4a86-9ef2-39587de0fc2e-kube-api-access-v2vnp\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:52.818246 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.818210    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv" event={"ID":"67155b07-975c-4a86-9ef2-39587de0fc2e","Type":"ContainerDied","Data":"f895e526f161800313dc960c9695c52aecebdd03f60d8f2124c28674d1199117"}
Nov 03 20:10:52.818717 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.818275    4531 scope.go:122] "RemoveContainer" containerID="19d92f501dbead2552582e16f4b69b6d164e3309d108220fc70d0cc1fda40fa0"
Nov 03 20:10:52.818717 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.818229    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"
Nov 03 20:10:52.839537 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.839510    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"]
Nov 03 20:10:52.847654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.847629    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zszqv"]
Nov 03 20:10:52.856697 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:52.856671    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67155b07-975c-4a86-9ef2-39587de0fc2e" path="/var/lib/kubelet/pods/67155b07-975c-4a86-9ef2-39587de0fc2e/volumes"
Nov 03 20:10:56.560788 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:56.560746    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:56.683102 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:56.683061    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:59.022333 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:59.022286    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"]
Nov 03 20:10:59.025244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:59.025218    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"]
Nov 03 20:10:59.029347 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:59.029319    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd71da9a-qz2gv"]
Nov 03 20:10:59.032500 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:10:59.032468    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a133ad1-58bjl"]
Nov 03 20:11:00.879199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:00.865567    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08310c0b-cba7-4218-bd85-3251607f6933" path="/var/lib/kubelet/pods/08310c0b-cba7-4218-bd85-3251607f6933/volumes"
Nov 03 20:11:00.880580 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:00.880544    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d317dc4c-09db-434c-be9b-923b1416e6af" path="/var/lib/kubelet/pods/d317dc4c-09db-434c-be9b-923b1416e6af/volumes"
Nov 03 20:11:11.021325 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:11.021233    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"]
Nov 03 20:11:11.023827 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:11.023798    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c54243-tcznc"]
Nov 03 20:11:11.026241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:11.026213    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"]
Nov 03 20:11:11.028216 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:11.028191    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b4b0eb7-2bl8v"]
Nov 03 20:11:12.857036 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.857002    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8737e64b-f721-496c-8e5e-b61f14e55c80" path="/var/lib/kubelet/pods/8737e64b-f721-496c-8e5e-b61f14e55c80/volumes"
Nov 03 20:11:12.857487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.857471    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9e3a33-2e59-4bbc-a431-fe482461a054" path="/var/lib/kubelet/pods/af9e3a33-2e59-4bbc-a431-fe482461a054/volumes"
Nov 03 20:11:12.923487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.923444    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"]
Nov 03 20:11:12.923900 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.923887    4531 cpu_manager.go:414] "containerMap: removing container" podUID="67155b07-975c-4a86-9ef2-39587de0fc2e" containerName="discovery"
Nov 03 20:11:12.923951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.923901    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67155b07-975c-4a86-9ef2-39587de0fc2e" containerName="discovery"
Nov 03 20:11:12.923951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.923914    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f761666e-87a9-4ff6-9309-490667dcd1c8" containerName="revoke-keys"
Nov 03 20:11:12.923951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.923919    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f761666e-87a9-4ff6-9309-490667dcd1c8" containerName="revoke-keys"
Nov 03 20:11:12.924058 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.924046    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="67155b07-975c-4a86-9ef2-39587de0fc2e" containerName="discovery"
Nov 03 20:11:12.924093 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.924058    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f761666e-87a9-4ff6-9309-490667dcd1c8" containerName="revoke-keys"
Nov 03 20:11:12.928503 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.928479    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:12.932258 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.932227    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:11:12.932454 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.932243    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:11:12.932454 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.932259    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:11:12.932632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.932311    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:11:12.932875 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.932855    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"]
Nov 03 20:11:12.942368 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.942339    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtpbb\" (UniqueName: \"kubernetes.io/projected/d241449b-a313-43ee-bb54-bea78b294161-kube-api-access-wtpbb\") pod \"maas-api-revoke-sub-046e1343-7tdjq\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:12.942526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.942382    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d241449b-a313-43ee-bb54-bea78b294161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7tdjq\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:12.942526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:12.942486    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d241449b-a313-43ee-bb54-bea78b294161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7tdjq\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:13.043951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:13.043911    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtpbb\" (UniqueName: \"kubernetes.io/projected/d241449b-a313-43ee-bb54-bea78b294161-kube-api-access-wtpbb\") pod \"maas-api-revoke-sub-046e1343-7tdjq\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:13.044152 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:13.043964    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d241449b-a313-43ee-bb54-bea78b294161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7tdjq\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:13.044152 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:13.044003    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d241449b-a313-43ee-bb54-bea78b294161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7tdjq\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:13.044705 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:13.044673    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d241449b-a313-43ee-bb54-bea78b294161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7tdjq\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:13.046595 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:13.046571    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d241449b-a313-43ee-bb54-bea78b294161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7tdjq\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:13.051136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:13.051098    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtpbb\" (UniqueName: \"kubernetes.io/projected/d241449b-a313-43ee-bb54-bea78b294161-kube-api-access-wtpbb\") pod \"maas-api-revoke-sub-046e1343-7tdjq\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:13.243873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:13.243818    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:13.529942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:13.529910    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"]
Nov 03 20:11:13.531965 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:11:13.531935    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd241449b_a313_43ee_bb54_bea78b294161.slice/crio-b97fe9e50148b6da304374ef1b8d58bf3df6984f6fccf9c330c37ae1e86a7d03 WatchSource:0}: Error finding container b97fe9e50148b6da304374ef1b8d58bf3df6984f6fccf9c330c37ae1e86a7d03: Status 404 returned error can't find the container with id b97fe9e50148b6da304374ef1b8d58bf3df6984f6fccf9c330c37ae1e86a7d03
Nov 03 20:11:13.995230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:13.995159    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq" event={"ID":"d241449b-a313-43ee-bb54-bea78b294161","Type":"ContainerStarted","Data":"5e71a7fd70fac3585736df5196abc8edbe7fc35da9a58ff08b4f81a46bde0e2a"}
Nov 03 20:11:13.995725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:13.995247    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq" event={"ID":"d241449b-a313-43ee-bb54-bea78b294161","Type":"ContainerStarted","Data":"b97fe9e50148b6da304374ef1b8d58bf3df6984f6fccf9c330c37ae1e86a7d03"}
Nov 03 20:11:14.009151 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:14.009092    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq" podStartSLOduration=2.009078068 podStartE2EDuration="2.009078068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:14.007818686 +0000 UTC m=+2387.277941982" watchObservedRunningTime="2026-11-03 20:11:14.009078068 +0000 UTC m=+2387.279201362"
Nov 03 20:11:15.003768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:15.003731    4531 generic.go:356] "Generic (PLEG): container finished" podID="d241449b-a313-43ee-bb54-bea78b294161" containerID="5e71a7fd70fac3585736df5196abc8edbe7fc35da9a58ff08b4f81a46bde0e2a" exitCode=0
Nov 03 20:11:15.004319 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:15.003809    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq" event={"ID":"d241449b-a313-43ee-bb54-bea78b294161","Type":"ContainerDied","Data":"5e71a7fd70fac3585736df5196abc8edbe7fc35da9a58ff08b4f81a46bde0e2a"}
Nov 03 20:11:16.146970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:16.146942    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:16.173226 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:16.173196    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d241449b-a313-43ee-bb54-bea78b294161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d241449b-a313-43ee-bb54-bea78b294161-maas-api-openshift-ca\") pod \"d241449b-a313-43ee-bb54-bea78b294161\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") "
Nov 03 20:11:16.173433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:16.173290    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d241449b-a313-43ee-bb54-bea78b294161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d241449b-a313-43ee-bb54-bea78b294161-maas-api-xks-ca\") pod \"d241449b-a313-43ee-bb54-bea78b294161\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") "
Nov 03 20:11:16.173433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:16.173324    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d241449b-a313-43ee-bb54-bea78b294161-kube-api-access-wtpbb\" (UniqueName: \"kubernetes.io/projected/d241449b-a313-43ee-bb54-bea78b294161-kube-api-access-wtpbb\") pod \"d241449b-a313-43ee-bb54-bea78b294161\" (UID: \"d241449b-a313-43ee-bb54-bea78b294161\") "
Nov 03 20:11:16.173629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:16.173603    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d241449b-a313-43ee-bb54-bea78b294161-maas-api-openshift-ca" pod "d241449b-a313-43ee-bb54-bea78b294161" (UID: "d241449b-a313-43ee-bb54-bea78b294161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:16.175380 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:16.175349    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d241449b-a313-43ee-bb54-bea78b294161-maas-api-xks-ca" pod "d241449b-a313-43ee-bb54-bea78b294161" (UID: "d241449b-a313-43ee-bb54-bea78b294161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:16.175506 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:16.175478    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d241449b-a313-43ee-bb54-bea78b294161-kube-api-access-wtpbb" pod "d241449b-a313-43ee-bb54-bea78b294161" (UID: "d241449b-a313-43ee-bb54-bea78b294161"). InnerVolumeSpecName "kube-api-access-wtpbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:16.274677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:16.274590    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d241449b-a313-43ee-bb54-bea78b294161-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:16.274677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:16.274624    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d241449b-a313-43ee-bb54-bea78b294161-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:16.274677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:16.274635    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtpbb\" (UniqueName: \"kubernetes.io/projected/d241449b-a313-43ee-bb54-bea78b294161-kube-api-access-wtpbb\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:17.017365 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:17.017329    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"
Nov 03 20:11:17.017365 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:17.017353    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq" event={"ID":"d241449b-a313-43ee-bb54-bea78b294161","Type":"ContainerDied","Data":"b97fe9e50148b6da304374ef1b8d58bf3df6984f6fccf9c330c37ae1e86a7d03"}
Nov 03 20:11:17.017605 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:17.017407    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b97fe9e50148b6da304374ef1b8d58bf3df6984f6fccf9c330c37ae1e86a7d03"
Nov 03 20:11:17.297511 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:17.297472    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:20.572156 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:11:20.572116    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:11:23.138022 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.137979    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"]
Nov 03 20:11:23.138519 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.138499    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d241449b-a313-43ee-bb54-bea78b294161" containerName="revoke-keys"
Nov 03 20:11:23.138565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.138521    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d241449b-a313-43ee-bb54-bea78b294161" containerName="revoke-keys"
Nov 03 20:11:23.138610 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.138585    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d241449b-a313-43ee-bb54-bea78b294161" containerName="revoke-keys"
Nov 03 20:11:23.141984 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.141963    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:23.145341 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.145308    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:11:23.145495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.145389    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:11:23.145495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.145413    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:11:23.145495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.145467    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:11:23.148111 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.148081    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"]
Nov 03 20:11:23.249829 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.249794    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-hbg9h\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:23.250028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.249856    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rv85\" (UniqueName: \"kubernetes.io/projected/70c54234-fb21-4d9c-b1eb-218db58da13b-kube-api-access-9rv85\") pod \"maas-api-revoke-sub-d3fba056-hbg9h\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:23.250028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.250005    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-hbg9h\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:23.351385 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.351344    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-hbg9h\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:23.351582 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.351477    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-hbg9h\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:23.351582 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.351533    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rv85\" (UniqueName: \"kubernetes.io/projected/70c54234-fb21-4d9c-b1eb-218db58da13b-kube-api-access-9rv85\") pod \"maas-api-revoke-sub-d3fba056-hbg9h\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:23.352134 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.352113    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-hbg9h\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:23.353860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.353841    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-hbg9h\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:23.358429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.358409    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rv85\" (UniqueName: \"kubernetes.io/projected/70c54234-fb21-4d9c-b1eb-218db58da13b-kube-api-access-9rv85\") pod \"maas-api-revoke-sub-d3fba056-hbg9h\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:23.461663 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.461565    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:23.736358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:23.736325    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"]
Nov 03 20:11:23.738595 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:11:23.738566    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c54234_fb21_4d9c_b1eb_218db58da13b.slice/crio-d7ec434d8ac319d2b58beb29238a82bd9ad89d6cb11c11cab262959b5e3f0ab5 WatchSource:0}: Error finding container d7ec434d8ac319d2b58beb29238a82bd9ad89d6cb11c11cab262959b5e3f0ab5: Status 404 returned error can't find the container with id d7ec434d8ac319d2b58beb29238a82bd9ad89d6cb11c11cab262959b5e3f0ab5
Nov 03 20:11:24.060052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:24.060020    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h" event={"ID":"70c54234-fb21-4d9c-b1eb-218db58da13b","Type":"ContainerStarted","Data":"ca25caed94a7fd1d22db213e42ab5c88acfd1a1e0c092471672f06d51ec497d4"}
Nov 03 20:11:24.060232 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:24.060079    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h" event={"ID":"70c54234-fb21-4d9c-b1eb-218db58da13b","Type":"ContainerStarted","Data":"d7ec434d8ac319d2b58beb29238a82bd9ad89d6cb11c11cab262959b5e3f0ab5"}
Nov 03 20:11:24.075114 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:24.075057    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h" podStartSLOduration=1.075044064 podStartE2EDuration="1.075044064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:24.072777525 +0000 UTC m=+2397.342900821" watchObservedRunningTime="2026-11-03 20:11:24.075044064 +0000 UTC m=+2397.345167349"
Nov 03 20:11:25.113828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:25.113723    4531 generic.go:356] "Generic (PLEG): container finished" podID="70c54234-fb21-4d9c-b1eb-218db58da13b" containerID="ca25caed94a7fd1d22db213e42ab5c88acfd1a1e0c092471672f06d51ec497d4" exitCode=0
Nov 03 20:11:25.114440 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:25.114412    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h" event={"ID":"70c54234-fb21-4d9c-b1eb-218db58da13b","Type":"ContainerDied","Data":"ca25caed94a7fd1d22db213e42ab5c88acfd1a1e0c092471672f06d51ec497d4"}
Nov 03 20:11:26.790024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:26.789986    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:26.878552 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:26.878521    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-xks-ca\") pod \"70c54234-fb21-4d9c-b1eb-218db58da13b\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") "
Nov 03 20:11:26.878777 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:26.878624    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c54234-fb21-4d9c-b1eb-218db58da13b-kube-api-access-9rv85\" (UniqueName: \"kubernetes.io/projected/70c54234-fb21-4d9c-b1eb-218db58da13b-kube-api-access-9rv85\") pod \"70c54234-fb21-4d9c-b1eb-218db58da13b\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") "
Nov 03 20:11:26.878777 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:26.878663    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-openshift-ca\") pod \"70c54234-fb21-4d9c-b1eb-218db58da13b\" (UID: \"70c54234-fb21-4d9c-b1eb-218db58da13b\") "
Nov 03 20:11:26.879253 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:26.879207    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-openshift-ca" pod "70c54234-fb21-4d9c-b1eb-218db58da13b" (UID: "70c54234-fb21-4d9c-b1eb-218db58da13b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:26.880950 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:26.880919    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c54234-fb21-4d9c-b1eb-218db58da13b-kube-api-access-9rv85" pod "70c54234-fb21-4d9c-b1eb-218db58da13b" (UID: "70c54234-fb21-4d9c-b1eb-218db58da13b"). InnerVolumeSpecName "kube-api-access-9rv85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:26.881573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:26.881547    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-xks-ca" pod "70c54234-fb21-4d9c-b1eb-218db58da13b" (UID: "70c54234-fb21-4d9c-b1eb-218db58da13b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:26.979894 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:26.979849    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rv85\" (UniqueName: \"kubernetes.io/projected/70c54234-fb21-4d9c-b1eb-218db58da13b-kube-api-access-9rv85\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:26.979894 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:26.979895    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:26.980130 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:26.979912    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c54234-fb21-4d9c-b1eb-218db58da13b-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:27.136989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:27.136957    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"
Nov 03 20:11:27.137186 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:27.136968    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h" event={"ID":"70c54234-fb21-4d9c-b1eb-218db58da13b","Type":"ContainerDied","Data":"d7ec434d8ac319d2b58beb29238a82bd9ad89d6cb11c11cab262959b5e3f0ab5"}
Nov 03 20:11:27.137186 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:27.137082    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7ec434d8ac319d2b58beb29238a82bd9ad89d6cb11c11cab262959b5e3f0ab5"
Nov 03 20:11:32.559637 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:32.559602    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:11:32.589916 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:32.589885    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:11:35.942573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:35.942538    4531 scope.go:122] "RemoveContainer" containerID="6d2d7bfb189aa1398bf761ede5e896f3aba8197bd0e313850e4214ab6c949e9b"
Nov 03 20:11:35.952327 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:35.952303    4531 scope.go:122] "RemoveContainer" containerID="d6bc67d8a54c11077373295037b210c075726e69f64211ceb34708e014336d0e"
Nov 03 20:11:35.961411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:35.961385    4531 scope.go:122] "RemoveContainer" containerID="0606ebb42f091d77a9e1a68cce0f56d6a6f2389863e7c90585ea07ebb799ca5f"
Nov 03 20:11:35.970074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:35.970055    4531 scope.go:122] "RemoveContainer" containerID="06b2824314de2741d868d0fa5b55cff24ffef052971970e779734b917e871af1"
Nov 03 20:11:35.978988 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:35.978969    4531 scope.go:122] "RemoveContainer" containerID="f82c622a18f371fff5d3e8c6c77fe29c569481d93db884d42c37ea2d463a138b"
Nov 03 20:11:35.987275 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:35.987257    4531 scope.go:122] "RemoveContainer" containerID="8b0f454328db6a4f527a04ce6b0f808c7c312b49d5248ede15d9f14886558bad"
Nov 03 20:11:35.996138 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:35.996121    4531 scope.go:122] "RemoveContainer" containerID="a097922824281914bcc21724af7047f53f1c4a896f812a582c0950e9ab41474d"
Nov 03 20:11:38.140198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:38.139333    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7n48w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:11:38.140198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:38.139442    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7n48w" podUID="bd352072-e76d-4d10-93a1-a71b2e0495ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:43.805880 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.805838    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"]
Nov 03 20:11:43.806467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.806447    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70c54234-fb21-4d9c-b1eb-218db58da13b" containerName="revoke-keys"
Nov 03 20:11:43.806521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.806470    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c54234-fb21-4d9c-b1eb-218db58da13b" containerName="revoke-keys"
Nov 03 20:11:43.806606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.806575    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c54234-fb21-4d9c-b1eb-218db58da13b" containerName="revoke-keys"
Nov 03 20:11:43.812669 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.812639    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:43.815132 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.815102    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"]
Nov 03 20:11:43.815776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.815738    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:11:43.815776 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.815770    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:11:43.816058 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.816032    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:11:43.816163 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.816095    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:11:43.898404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.898363    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88f2\" (UniqueName: \"kubernetes.io/projected/9afa88be-a289-4c8f-b6cc-a370296e1b60-kube-api-access-p88f2\") pod \"maas-api-revoke-sub-ae594979-9952m\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:43.898404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.898404    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9952m\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:43.898691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.898617    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9952m\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:43.999282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.999248    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88f2\" (UniqueName: \"kubernetes.io/projected/9afa88be-a289-4c8f-b6cc-a370296e1b60-kube-api-access-p88f2\") pod \"maas-api-revoke-sub-ae594979-9952m\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:43.999282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.999287    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9952m\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:43.999562 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:43.999434    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9952m\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:44.000427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:44.000403    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9952m\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:44.002299 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:44.002278    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9952m\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:44.006227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:44.006204    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88f2\" (UniqueName: \"kubernetes.io/projected/9afa88be-a289-4c8f-b6cc-a370296e1b60-kube-api-access-p88f2\") pod \"maas-api-revoke-sub-ae594979-9952m\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:44.127955 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:44.127848    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:44.401195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:44.401148    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"]
Nov 03 20:11:44.403327 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:11:44.403291    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afa88be_a289_4c8f_b6cc_a370296e1b60.slice/crio-2ce9fb65bd44c74744f18611fc78e2ae239ce61dac431bc6c17d7343922c49fe WatchSource:0}: Error finding container 2ce9fb65bd44c74744f18611fc78e2ae239ce61dac431bc6c17d7343922c49fe: Status 404 returned error can't find the container with id 2ce9fb65bd44c74744f18611fc78e2ae239ce61dac431bc6c17d7343922c49fe
Nov 03 20:11:45.272250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:45.272215    4531 generic.go:356] "Generic (PLEG): container finished" podID="9afa88be-a289-4c8f-b6cc-a370296e1b60" containerID="58d9c1ea42392f07ac8d9449c5a01bbcd85ceb8c72b7fa167b48eb8e16ca2293" exitCode=0
Nov 03 20:11:45.272653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:45.272295    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m" event={"ID":"9afa88be-a289-4c8f-b6cc-a370296e1b60","Type":"ContainerDied","Data":"58d9c1ea42392f07ac8d9449c5a01bbcd85ceb8c72b7fa167b48eb8e16ca2293"}
Nov 03 20:11:45.272653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:45.272349    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m" event={"ID":"9afa88be-a289-4c8f-b6cc-a370296e1b60","Type":"ContainerStarted","Data":"2ce9fb65bd44c74744f18611fc78e2ae239ce61dac431bc6c17d7343922c49fe"}
Nov 03 20:11:46.408299 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:46.408273    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:46.526269 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:46.526230    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-openshift-ca\") pod \"9afa88be-a289-4c8f-b6cc-a370296e1b60\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") "
Nov 03 20:11:46.526474 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:46.526376    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afa88be-a289-4c8f-b6cc-a370296e1b60-kube-api-access-p88f2\" (UniqueName: \"kubernetes.io/projected/9afa88be-a289-4c8f-b6cc-a370296e1b60-kube-api-access-p88f2\") pod \"9afa88be-a289-4c8f-b6cc-a370296e1b60\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") "
Nov 03 20:11:46.526474 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:46.526408    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-xks-ca\") pod \"9afa88be-a289-4c8f-b6cc-a370296e1b60\" (UID: \"9afa88be-a289-4c8f-b6cc-a370296e1b60\") "
Nov 03 20:11:46.526677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:46.526647    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-openshift-ca" pod "9afa88be-a289-4c8f-b6cc-a370296e1b60" (UID: "9afa88be-a289-4c8f-b6cc-a370296e1b60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:46.528654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:46.528599    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-xks-ca" pod "9afa88be-a289-4c8f-b6cc-a370296e1b60" (UID: "9afa88be-a289-4c8f-b6cc-a370296e1b60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:46.528654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:46.528606    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afa88be-a289-4c8f-b6cc-a370296e1b60-kube-api-access-p88f2" pod "9afa88be-a289-4c8f-b6cc-a370296e1b60" (UID: "9afa88be-a289-4c8f-b6cc-a370296e1b60"). InnerVolumeSpecName "kube-api-access-p88f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:46.627502 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:46.627468    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p88f2\" (UniqueName: \"kubernetes.io/projected/9afa88be-a289-4c8f-b6cc-a370296e1b60-kube-api-access-p88f2\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:46.627502 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:46.627500    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:46.627502 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:46.627510    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afa88be-a289-4c8f-b6cc-a370296e1b60-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:47.286206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:47.286156    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m" event={"ID":"9afa88be-a289-4c8f-b6cc-a370296e1b60","Type":"ContainerDied","Data":"2ce9fb65bd44c74744f18611fc78e2ae239ce61dac431bc6c17d7343922c49fe"}
Nov 03 20:11:47.286206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:47.286191    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"
Nov 03 20:11:47.286433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:47.286222    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ce9fb65bd44c74744f18611fc78e2ae239ce61dac431bc6c17d7343922c49fe"
Nov 03 20:11:50.572766 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:11:50.572719    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:11:53.024442 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:53.024406    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"]
Nov 03 20:11:53.028264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:53.028232    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"]
Nov 03 20:11:53.029526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:53.029505    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f913748c-4wx7q"]
Nov 03 20:11:53.031480 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:53.031455    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83e3d217-72xjc"]
Nov 03 20:11:54.857262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:54.857227    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a77b64-e0cd-4766-a0f1-e0b25cf73e54" path="/var/lib/kubelet/pods/75a77b64-e0cd-4766-a0f1-e0b25cf73e54/volumes"
Nov 03 20:11:54.857805 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:11:54.857789    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e77260e-a4d1-4097-9859-8ece56379fd0" path="/var/lib/kubelet/pods/7e77260e-a4d1-4097-9859-8ece56379fd0/volumes"
Nov 03 20:12:01.939657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:01.939613    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"]
Nov 03 20:12:01.940086 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:01.940048    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9afa88be-a289-4c8f-b6cc-a370296e1b60" containerName="revoke-keys"
Nov 03 20:12:01.940086 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:01.940059    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afa88be-a289-4c8f-b6cc-a370296e1b60" containerName="revoke-keys"
Nov 03 20:12:01.940200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:01.940119    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afa88be-a289-4c8f-b6cc-a370296e1b60" containerName="revoke-keys"
Nov 03 20:12:01.943754 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:01.943732    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:01.946617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:01.946590    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:12:01.946809 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:01.946691    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:12:01.947464 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:01.947442    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:12:01.947722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:01.947704    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:12:01.949519 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:01.949489    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"]
Nov 03 20:12:02.042695 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:02.042657    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hjbpw\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:02.042695 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:02.042696    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvvhs\" (UniqueName: \"kubernetes.io/projected/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-kube-api-access-cvvhs\") pod \"maas-api-revoke-sub-a4aef7d7-hjbpw\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:02.042929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:02.042804    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hjbpw\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:02.143839 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:02.143801    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hjbpw\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:02.143839 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:02.143840    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvvhs\" (UniqueName: \"kubernetes.io/projected/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-kube-api-access-cvvhs\") pod \"maas-api-revoke-sub-a4aef7d7-hjbpw\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:02.144087 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:02.143884    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hjbpw\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:02.144562 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:02.144536    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hjbpw\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:02.146406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:02.146379    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hjbpw\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:02.151009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:02.150983    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvvhs\" (UniqueName: \"kubernetes.io/projected/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-kube-api-access-cvvhs\") pod \"maas-api-revoke-sub-a4aef7d7-hjbpw\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:02.260002 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:02.259910    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:02.540979 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:02.540944    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"]
Nov 03 20:12:02.542649 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:12:02.542613    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d39ddc_600d_4d84_9ca8_42144a6e6b0d.slice/crio-80da4c1f844dae8acd6b116ed8354299afa3b77d45e304ac7e6d0c006acf4d52 WatchSource:0}: Error finding container 80da4c1f844dae8acd6b116ed8354299afa3b77d45e304ac7e6d0c006acf4d52: Status 404 returned error can't find the container with id 80da4c1f844dae8acd6b116ed8354299afa3b77d45e304ac7e6d0c006acf4d52
Nov 03 20:12:03.400852 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:03.400816    4531 generic.go:356] "Generic (PLEG): container finished" podID="31d39ddc-600d-4d84-9ca8-42144a6e6b0d" containerID="56470292fefbaf8a204433bd3978c9e1e78ee08ef28e1b5b58bd9b77755cbc52" exitCode=0
Nov 03 20:12:03.401423 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:03.400913    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw" event={"ID":"31d39ddc-600d-4d84-9ca8-42144a6e6b0d","Type":"ContainerDied","Data":"56470292fefbaf8a204433bd3978c9e1e78ee08ef28e1b5b58bd9b77755cbc52"}
Nov 03 20:12:03.401423 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:03.400988    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw" event={"ID":"31d39ddc-600d-4d84-9ca8-42144a6e6b0d","Type":"ContainerStarted","Data":"80da4c1f844dae8acd6b116ed8354299afa3b77d45e304ac7e6d0c006acf4d52"}
Nov 03 20:12:04.540748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:04.540721    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:04.669896 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:04.669805    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-openshift-ca\") pod \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") "
Nov 03 20:12:04.670077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:04.669910    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-xks-ca\") pod \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") "
Nov 03 20:12:04.670077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:04.669940    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-kube-api-access-cvvhs\" (UniqueName: \"kubernetes.io/projected/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-kube-api-access-cvvhs\") pod \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\" (UID: \"31d39ddc-600d-4d84-9ca8-42144a6e6b0d\") "
Nov 03 20:12:04.670239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:04.670217    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-openshift-ca" pod "31d39ddc-600d-4d84-9ca8-42144a6e6b0d" (UID: "31d39ddc-600d-4d84-9ca8-42144a6e6b0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:04.672209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:04.672182    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-xks-ca" pod "31d39ddc-600d-4d84-9ca8-42144a6e6b0d" (UID: "31d39ddc-600d-4d84-9ca8-42144a6e6b0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:04.672339 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:04.672223    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-kube-api-access-cvvhs" pod "31d39ddc-600d-4d84-9ca8-42144a6e6b0d" (UID: "31d39ddc-600d-4d84-9ca8-42144a6e6b0d"). InnerVolumeSpecName "kube-api-access-cvvhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:04.771749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:04.771706    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:04.771749 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:04.771751    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:04.772011 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:04.771765    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvvhs\" (UniqueName: \"kubernetes.io/projected/31d39ddc-600d-4d84-9ca8-42144a6e6b0d-kube-api-access-cvvhs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:05.412609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:05.412572    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"
Nov 03 20:12:05.412833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:05.412605    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw" event={"ID":"31d39ddc-600d-4d84-9ca8-42144a6e6b0d","Type":"ContainerDied","Data":"80da4c1f844dae8acd6b116ed8354299afa3b77d45e304ac7e6d0c006acf4d52"}
Nov 03 20:12:05.412833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:05.412655    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80da4c1f844dae8acd6b116ed8354299afa3b77d45e304ac7e6d0c006acf4d52"
Nov 03 20:12:13.027569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:13.027523    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"]
Nov 03 20:12:13.028120 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:13.027595    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b23a01-7vpsx"]
Nov 03 20:12:14.264295 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:14.264262    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:14.857201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:14.857154    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6ddc7e-43da-425e-aca3-22a796701df4" path="/var/lib/kubelet/pods/fd6ddc7e-43da-425e-aca3-22a796701df4/volumes"
Nov 03 20:12:17.320973 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:17.320476    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:20.574062 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:12:20.574008    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:12:20.624538 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.624508    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"]
Nov 03 20:12:20.624941 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.624928    4531 cpu_manager.go:414] "containerMap: removing container" podUID="31d39ddc-600d-4d84-9ca8-42144a6e6b0d" containerName="revoke-keys"
Nov 03 20:12:20.624997 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.624942    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d39ddc-600d-4d84-9ca8-42144a6e6b0d" containerName="revoke-keys"
Nov 03 20:12:20.625040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.625035    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d39ddc-600d-4d84-9ca8-42144a6e6b0d" containerName="revoke-keys"
Nov 03 20:12:20.628667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.628648    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:20.631883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.631861    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:12:20.632363 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.632339    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:12:20.632679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.632609    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:12:20.635955 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.635930    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"]
Nov 03 20:12:20.636063 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.636004    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:12:20.664920 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.664887    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9slrb\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:20.665105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.664953    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9slrb\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:20.665105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.665039    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kfc7\" (UniqueName: \"kubernetes.io/projected/8f074e13-c713-4e39-8780-b452c3c8b176-kube-api-access-9kfc7\") pod \"maas-api-revoke-sub-79805ca9-9slrb\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:20.766312 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.766265    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9slrb\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:20.766479 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.766356    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9slrb\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:20.766479 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.766398    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kfc7\" (UniqueName: \"kubernetes.io/projected/8f074e13-c713-4e39-8780-b452c3c8b176-kube-api-access-9kfc7\") pod \"maas-api-revoke-sub-79805ca9-9slrb\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:20.767230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.767206    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-9slrb\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:20.768785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.768762    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-9slrb\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:20.773426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.773406    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kfc7\" (UniqueName: \"kubernetes.io/projected/8f074e13-c713-4e39-8780-b452c3c8b176-kube-api-access-9kfc7\") pod \"maas-api-revoke-sub-79805ca9-9slrb\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:20.944151 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:20.944110    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:21.222157 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:21.222123    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"]
Nov 03 20:12:21.223699 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:12:21.223670    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f074e13_c713_4e39_8780_b452c3c8b176.slice/crio-7d145a7fe15d48f34e1cb88f78ec857283126c0bb83ab89a791c6541d16aa454 WatchSource:0}: Error finding container 7d145a7fe15d48f34e1cb88f78ec857283126c0bb83ab89a791c6541d16aa454: Status 404 returned error can't find the container with id 7d145a7fe15d48f34e1cb88f78ec857283126c0bb83ab89a791c6541d16aa454
Nov 03 20:12:21.534656 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:21.534568    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb" event={"ID":"8f074e13-c713-4e39-8780-b452c3c8b176","Type":"ContainerStarted","Data":"7d145a7fe15d48f34e1cb88f78ec857283126c0bb83ab89a791c6541d16aa454"}
Nov 03 20:12:22.541532 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:22.541443    4531 generic.go:356] "Generic (PLEG): container finished" podID="8f074e13-c713-4e39-8780-b452c3c8b176" containerID="857e6cadaf870b7b9125f2c76f166ba87e7a0b2e05a5958161f4d7655f6db543" exitCode=0
Nov 03 20:12:22.541908 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:22.541527    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb" event={"ID":"8f074e13-c713-4e39-8780-b452c3c8b176","Type":"ContainerDied","Data":"857e6cadaf870b7b9125f2c76f166ba87e7a0b2e05a5958161f4d7655f6db543"}
Nov 03 20:12:23.020900 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.020847    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"]
Nov 03 20:12:23.023213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.023186    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e710c56-9tkxf"]
Nov 03 20:12:23.679667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.679638    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:23.796111 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.796072    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-openshift-ca\") pod \"8f074e13-c713-4e39-8780-b452c3c8b176\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") "
Nov 03 20:12:23.796324 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.796130    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-xks-ca\") pod \"8f074e13-c713-4e39-8780-b452c3c8b176\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") "
Nov 03 20:12:23.796324 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.796290    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f074e13-c713-4e39-8780-b452c3c8b176-kube-api-access-9kfc7\" (UniqueName: \"kubernetes.io/projected/8f074e13-c713-4e39-8780-b452c3c8b176-kube-api-access-9kfc7\") pod \"8f074e13-c713-4e39-8780-b452c3c8b176\" (UID: \"8f074e13-c713-4e39-8780-b452c3c8b176\") "
Nov 03 20:12:23.796554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.796522    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-openshift-ca" pod "8f074e13-c713-4e39-8780-b452c3c8b176" (UID: "8f074e13-c713-4e39-8780-b452c3c8b176"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:23.798298 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.798236    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-xks-ca" pod "8f074e13-c713-4e39-8780-b452c3c8b176" (UID: "8f074e13-c713-4e39-8780-b452c3c8b176"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:23.798298 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.798291    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f074e13-c713-4e39-8780-b452c3c8b176-kube-api-access-9kfc7" pod "8f074e13-c713-4e39-8780-b452c3c8b176" (UID: "8f074e13-c713-4e39-8780-b452c3c8b176"). InnerVolumeSpecName "kube-api-access-9kfc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:23.897914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.897873    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kfc7\" (UniqueName: \"kubernetes.io/projected/8f074e13-c713-4e39-8780-b452c3c8b176-kube-api-access-9kfc7\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:23.897914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.897906    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:23.897914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:23.897916    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f074e13-c713-4e39-8780-b452c3c8b176-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:24.557304 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:24.557265    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb" event={"ID":"8f074e13-c713-4e39-8780-b452c3c8b176","Type":"ContainerDied","Data":"7d145a7fe15d48f34e1cb88f78ec857283126c0bb83ab89a791c6541d16aa454"}
Nov 03 20:12:24.557497 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:24.557309    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"
Nov 03 20:12:24.557497 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:24.557348    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d145a7fe15d48f34e1cb88f78ec857283126c0bb83ab89a791c6541d16aa454"
Nov 03 20:12:24.857377 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:24.857297    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d621c61-9042-4275-a9e1-88f2af78f655" path="/var/lib/kubelet/pods/0d621c61-9042-4275-a9e1-88f2af78f655/volumes"
Nov 03 20:12:29.020817 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:29.020778    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"]
Nov 03 20:12:29.023313 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:29.023286    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785aeb92-tkc7b"]
Nov 03 20:12:30.856657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:30.856618    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1441a6f-426c-4869-a3da-b0b0be8c0d6d" path="/var/lib/kubelet/pods/d1441a6f-426c-4869-a3da-b0b0be8c0d6d/volumes"
Nov 03 20:12:36.009310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:36.009266    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tdppc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:12:36.010117 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:36.010078    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" podUID="de3afce5-6597-4f19-8f0f-d4a695f4af39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:12:36.059326 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:36.059283    4531 scope.go:122] "RemoveContainer" containerID="b8030e363e398b89ed14fbc7c2ffe1884b90103b57be198597f94bfc37276c8a"
Nov 03 20:12:36.097272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:36.097227    4531 scope.go:122] "RemoveContainer" containerID="4c3d70cd5af568c724d16d3df4cb7c127e2262fd35eb44e0d9ca9de9aaa6ee51"
Nov 03 20:12:36.121946 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:36.121920    4531 scope.go:122] "RemoveContainer" containerID="954da0ed07416f83e03d229b4b87e32bda236804c75a9b189155c894751c8097"
Nov 03 20:12:36.160960 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:36.160931    4531 scope.go:122] "RemoveContainer" containerID="5e273430866b7964e8cf541eabf56a381bbf4545dd5afee50b8ca07dd4590f45"
Nov 03 20:12:36.216345 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:36.216310    4531 scope.go:122] "RemoveContainer" containerID="aae25fd63be1f917299e25fd979c39e3573d674a10b2a9f72f064d8b8c37d8e8"
Nov 03 20:12:36.234054 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:36.234030    4531 scope.go:122] "RemoveContainer" containerID="7edc56e526224d83fb6fd53968fffb84fa2fc1505168f7cba3c5b226fb2b9c2b"
Nov 03 20:12:36.274430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:36.271357    4531 scope.go:122] "RemoveContainer" containerID="26e15ca5c9aecabd4a1aff4da8ddcd3681b522af59407f06624eff5b06cc9479"
Nov 03 20:12:38.314542 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.314506    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"]
Nov 03 20:12:38.314964 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.314949    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8f074e13-c713-4e39-8780-b452c3c8b176" containerName="revoke-keys"
Nov 03 20:12:38.315010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.314965    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f074e13-c713-4e39-8780-b452c3c8b176" containerName="revoke-keys"
Nov 03 20:12:38.315067 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.315042    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f074e13-c713-4e39-8780-b452c3c8b176" containerName="revoke-keys"
Nov 03 20:12:38.320553 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.320529    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:38.324567 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.324443    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:12:38.324567 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.324512    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:12:38.324786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.324632    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:12:38.324786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.324720    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:12:38.326009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.325971    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"]
Nov 03 20:12:38.395431 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.395395    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k9sxr\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:38.395431 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.395430    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkbx\" (UniqueName: \"kubernetes.io/projected/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-kube-api-access-6gkbx\") pod \"maas-api-revoke-sub-0b7e8edb-k9sxr\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:38.395653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.395570    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k9sxr\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:38.496496 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.496457    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k9sxr\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:38.496708 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.496550    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k9sxr\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:38.496708 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.496568    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkbx\" (UniqueName: \"kubernetes.io/projected/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-kube-api-access-6gkbx\") pod \"maas-api-revoke-sub-0b7e8edb-k9sxr\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:38.497465 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.497396    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k9sxr\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:38.499058 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.499033    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-k9sxr\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:38.503336 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.503309    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkbx\" (UniqueName: \"kubernetes.io/projected/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-kube-api-access-6gkbx\") pod \"maas-api-revoke-sub-0b7e8edb-k9sxr\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:38.638960 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.638928    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:38.913728 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:38.913698    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"]
Nov 03 20:12:38.915481 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:12:38.915456    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde72b837_9c6a_4efc_b11f_ccf378eb7f8f.slice/crio-eb8c72e6927304812f9cc53b87e9aee32c09c863b6369089ac2fa933a0f9be3f WatchSource:0}: Error finding container eb8c72e6927304812f9cc53b87e9aee32c09c863b6369089ac2fa933a0f9be3f: Status 404 returned error can't find the container with id eb8c72e6927304812f9cc53b87e9aee32c09c863b6369089ac2fa933a0f9be3f
Nov 03 20:12:39.020037 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:39.020003    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"]
Nov 03 20:12:39.022233 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:39.022210    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c812237a-fwlxz"]
Nov 03 20:12:39.751421 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:39.751380    4531 generic.go:356] "Generic (PLEG): container finished" podID="de72b837-9c6a-4efc-b11f-ccf378eb7f8f" containerID="f782e95f242d19a7b7d6f7b6f62798eb432abb82ff00c355d97b0c415cb1f6ce" exitCode=0
Nov 03 20:12:39.751847 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:39.751464    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr" event={"ID":"de72b837-9c6a-4efc-b11f-ccf378eb7f8f","Type":"ContainerDied","Data":"f782e95f242d19a7b7d6f7b6f62798eb432abb82ff00c355d97b0c415cb1f6ce"}
Nov 03 20:12:39.751847 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:39.751528    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr" event={"ID":"de72b837-9c6a-4efc-b11f-ccf378eb7f8f","Type":"ContainerStarted","Data":"eb8c72e6927304812f9cc53b87e9aee32c09c863b6369089ac2fa933a0f9be3f"}
Nov 03 20:12:40.857438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:40.857407    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1f628d-2a51-4bf7-a162-a800effecda7" path="/var/lib/kubelet/pods/8d1f628d-2a51-4bf7-a162-a800effecda7/volumes"
Nov 03 20:12:40.886951 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:40.886924    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:40.919797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:40.919757    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-kube-api-access-6gkbx\" (UniqueName: \"kubernetes.io/projected/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-kube-api-access-6gkbx\") pod \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") "
Nov 03 20:12:40.919976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:40.919955    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-openshift-ca\") pod \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") "
Nov 03 20:12:40.920038 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:40.919993    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-xks-ca\") pod \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\" (UID: \"de72b837-9c6a-4efc-b11f-ccf378eb7f8f\") "
Nov 03 20:12:40.920358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:40.920333    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-openshift-ca" pod "de72b837-9c6a-4efc-b11f-ccf378eb7f8f" (UID: "de72b837-9c6a-4efc-b11f-ccf378eb7f8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:40.921905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:40.921883    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-kube-api-access-6gkbx" pod "de72b837-9c6a-4efc-b11f-ccf378eb7f8f" (UID: "de72b837-9c6a-4efc-b11f-ccf378eb7f8f"). InnerVolumeSpecName "kube-api-access-6gkbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:40.922037 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:40.922021    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-xks-ca" pod "de72b837-9c6a-4efc-b11f-ccf378eb7f8f" (UID: "de72b837-9c6a-4efc-b11f-ccf378eb7f8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:41.021864 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:41.021752    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gkbx\" (UniqueName: \"kubernetes.io/projected/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-kube-api-access-6gkbx\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:41.021864 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:41.021798    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:41.021864 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:41.021814    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de72b837-9c6a-4efc-b11f-ccf378eb7f8f-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:41.763099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:41.763066    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr" event={"ID":"de72b837-9c6a-4efc-b11f-ccf378eb7f8f","Type":"ContainerDied","Data":"eb8c72e6927304812f9cc53b87e9aee32c09c863b6369089ac2fa933a0f9be3f"}
Nov 03 20:12:41.763099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:41.763096    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"
Nov 03 20:12:41.763356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:41.763116    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb8c72e6927304812f9cc53b87e9aee32c09c863b6369089ac2fa933a0f9be3f"
Nov 03 20:12:48.961875 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:48.961835    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"]
Nov 03 20:12:48.962537 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:48.962516    4531 cpu_manager.go:414] "containerMap: removing container" podUID="de72b837-9c6a-4efc-b11f-ccf378eb7f8f" containerName="revoke-keys"
Nov 03 20:12:48.962587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:48.962541    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de72b837-9c6a-4efc-b11f-ccf378eb7f8f" containerName="revoke-keys"
Nov 03 20:12:48.962653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:48.962639    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="de72b837-9c6a-4efc-b11f-ccf378eb7f8f" containerName="revoke-keys"
Nov 03 20:12:48.966449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:48.966426    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:48.970522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:48.970478    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"]
Nov 03 20:12:48.972451 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:48.972427    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:12:48.973140 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:48.973119    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:12:48.973275 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:48.973162    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:12:48.973275 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:48.973210    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:12:49.101571 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.101536    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrd6k\" (UniqueName: \"kubernetes.io/projected/097f090f-a81f-41fd-a776-979eef4fe1aa-kube-api-access-vrd6k\") pod \"maas-api-revoke-sub-3d4156f1-q495m\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:49.101571 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.101575    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-q495m\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:49.101816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.101595    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-q495m\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:49.202915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.202875    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrd6k\" (UniqueName: \"kubernetes.io/projected/097f090f-a81f-41fd-a776-979eef4fe1aa-kube-api-access-vrd6k\") pod \"maas-api-revoke-sub-3d4156f1-q495m\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:49.202915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.202921    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-q495m\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:49.203241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.202943    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-q495m\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:49.203716 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.203689    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-q495m\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:49.205321 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.205303    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-q495m\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:49.209979 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.209957    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrd6k\" (UniqueName: \"kubernetes.io/projected/097f090f-a81f-41fd-a776-979eef4fe1aa-kube-api-access-vrd6k\") pod \"maas-api-revoke-sub-3d4156f1-q495m\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:49.285166 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.285083    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:49.556836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.556808    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"]
Nov 03 20:12:49.558947 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:12:49.558910    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097f090f_a81f_41fd_a776_979eef4fe1aa.slice/crio-70c69f15e9a2c5641f07f1782012527f4b2e645c14840dc582815910601643df WatchSource:0}: Error finding container 70c69f15e9a2c5641f07f1782012527f4b2e645c14840dc582815910601643df: Status 404 returned error can't find the container with id 70c69f15e9a2c5641f07f1782012527f4b2e645c14840dc582815910601643df
Nov 03 20:12:49.814797 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:49.814701    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m" event={"ID":"097f090f-a81f-41fd-a776-979eef4fe1aa","Type":"ContainerStarted","Data":"70c69f15e9a2c5641f07f1782012527f4b2e645c14840dc582815910601643df"}
Nov 03 20:12:50.573227 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:12:50.572675    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:12:50.821793 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:50.821702    4531 generic.go:356] "Generic (PLEG): container finished" podID="097f090f-a81f-41fd-a776-979eef4fe1aa" containerID="17b8c1e5c0b32ba33a970b8d88b72da23fbbc3af4e951a3c25b71fd47f046fe2" exitCode=0
Nov 03 20:12:50.821976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:50.821790    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m" event={"ID":"097f090f-a81f-41fd-a776-979eef4fe1aa","Type":"ContainerDied","Data":"17b8c1e5c0b32ba33a970b8d88b72da23fbbc3af4e951a3c25b71fd47f046fe2"}
Nov 03 20:12:51.962630 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:51.962603    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:52.035079 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.035045    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-xks-ca\") pod \"097f090f-a81f-41fd-a776-979eef4fe1aa\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") "
Nov 03 20:12:52.035293 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.035144    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-openshift-ca\") pod \"097f090f-a81f-41fd-a776-979eef4fe1aa\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") "
Nov 03 20:12:52.035293 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.035278    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097f090f-a81f-41fd-a776-979eef4fe1aa-kube-api-access-vrd6k\" (UniqueName: \"kubernetes.io/projected/097f090f-a81f-41fd-a776-979eef4fe1aa-kube-api-access-vrd6k\") pod \"097f090f-a81f-41fd-a776-979eef4fe1aa\" (UID: \"097f090f-a81f-41fd-a776-979eef4fe1aa\") "
Nov 03 20:12:52.035585 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.035558    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-openshift-ca" pod "097f090f-a81f-41fd-a776-979eef4fe1aa" (UID: "097f090f-a81f-41fd-a776-979eef4fe1aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:52.035693 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.035678    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:52.037368 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.037342    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097f090f-a81f-41fd-a776-979eef4fe1aa-kube-api-access-vrd6k" pod "097f090f-a81f-41fd-a776-979eef4fe1aa" (UID: "097f090f-a81f-41fd-a776-979eef4fe1aa"). InnerVolumeSpecName "kube-api-access-vrd6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:52.037477 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.037368    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-xks-ca" pod "097f090f-a81f-41fd-a776-979eef4fe1aa" (UID: "097f090f-a81f-41fd-a776-979eef4fe1aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:52.136209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.136157    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097f090f-a81f-41fd-a776-979eef4fe1aa-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:52.136363 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.136213    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrd6k\" (UniqueName: \"kubernetes.io/projected/097f090f-a81f-41fd-a776-979eef4fe1aa-kube-api-access-vrd6k\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:52.834158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.834122    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"
Nov 03 20:12:52.834375 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.834157    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m" event={"ID":"097f090f-a81f-41fd-a776-979eef4fe1aa","Type":"ContainerDied","Data":"70c69f15e9a2c5641f07f1782012527f4b2e645c14840dc582815910601643df"}
Nov 03 20:12:52.834375 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:12:52.834225    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70c69f15e9a2c5641f07f1782012527f4b2e645c14840dc582815910601643df"
Nov 03 20:13:06.991392 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:06.991353    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"]
Nov 03 20:13:06.994892 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:06.994862    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-q495m"]
Nov 03 20:13:07.017493 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.017456    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"]
Nov 03 20:13:07.017928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.017914    4531 cpu_manager.go:414] "containerMap: removing container" podUID="097f090f-a81f-41fd-a776-979eef4fe1aa" containerName="revoke-keys"
Nov 03 20:13:07.017985 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.017929    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097f090f-a81f-41fd-a776-979eef4fe1aa" containerName="revoke-keys"
Nov 03 20:13:07.018028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.018011    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="097f090f-a81f-41fd-a776-979eef4fe1aa" containerName="revoke-keys"
Nov 03 20:13:07.021207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.021186    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:07.025405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.025141    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:13:07.025405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.025270    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:13:07.025405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.025318    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:13:07.025405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.025319    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:13:07.027681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.027655    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"]
Nov 03 20:13:07.083666 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.083617    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmjb\" (UniqueName: \"kubernetes.io/projected/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-kube-api-access-rtmjb\") pod \"maas-api-revoke-sub-3d4156f1-9tttz\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:07.083856 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.083687    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9tttz\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:07.083856 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.083787    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9tttz\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:07.185355 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.185315    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9tttz\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:07.185569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.185397    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmjb\" (UniqueName: \"kubernetes.io/projected/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-kube-api-access-rtmjb\") pod \"maas-api-revoke-sub-3d4156f1-9tttz\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:07.185569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.185433    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9tttz\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:07.186112 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.186090    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9tttz\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:07.187884 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.187864    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9tttz\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:07.192349 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.192326    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmjb\" (UniqueName: \"kubernetes.io/projected/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-kube-api-access-rtmjb\") pod \"maas-api-revoke-sub-3d4156f1-9tttz\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:07.341612 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.341508    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:07.615832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.615594    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"]
Nov 03 20:13:07.618144 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:13:07.618117    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8367c5e_a1b6_476b_9f31_1a1fc1b29bd4.slice/crio-9e97f8629668ddf0a440bdb506c7dc1a737cc3545473d251944670f479035222 WatchSource:0}: Error finding container 9e97f8629668ddf0a440bdb506c7dc1a737cc3545473d251944670f479035222: Status 404 returned error can't find the container with id 9e97f8629668ddf0a440bdb506c7dc1a737cc3545473d251944670f479035222
Nov 03 20:13:07.966997 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.966960    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz" event={"ID":"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4","Type":"ContainerStarted","Data":"acd8eb9dcf8976c65c105742a7f1eb9afe69099e714110e2350d3323c7f3c632"}
Nov 03 20:13:07.967181 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.967023    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz" event={"ID":"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4","Type":"ContainerStarted","Data":"9e97f8629668ddf0a440bdb506c7dc1a737cc3545473d251944670f479035222"}
Nov 03 20:13:07.981286 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:07.980987    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz" podStartSLOduration=0.980968906 podStartE2EDuration="980.968906ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:07.979167024 +0000 UTC m=+2501.249290312" watchObservedRunningTime="2026-11-03 20:13:07.980968906 +0000 UTC m=+2501.251092203"
Nov 03 20:13:08.856912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:08.856827    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="097f090f-a81f-41fd-a776-979eef4fe1aa" path="/var/lib/kubelet/pods/097f090f-a81f-41fd-a776-979eef4fe1aa/volumes"
Nov 03 20:13:08.974189 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:08.974148    4531 generic.go:356] "Generic (PLEG): container finished" podID="f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4" containerID="acd8eb9dcf8976c65c105742a7f1eb9afe69099e714110e2350d3323c7f3c632" exitCode=0
Nov 03 20:13:08.974366 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:08.974234    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz" event={"ID":"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4","Type":"ContainerDied","Data":"acd8eb9dcf8976c65c105742a7f1eb9afe69099e714110e2350d3323c7f3c632"}
Nov 03 20:13:10.110127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.110099    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:10.214272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.214233    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-xks-ca\") pod \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") "
Nov 03 20:13:10.214473 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.214314    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-openshift-ca\") pod \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") "
Nov 03 20:13:10.214473 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.214410    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-kube-api-access-rtmjb\" (UniqueName: \"kubernetes.io/projected/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-kube-api-access-rtmjb\") pod \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\" (UID: \"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4\") "
Nov 03 20:13:10.214644 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.214628    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-openshift-ca" pod "f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4" (UID: "f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:10.214805 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.214777    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:10.216495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.216475    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-xks-ca" pod "f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4" (UID: "f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:13:10.216558 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.216534    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-kube-api-access-rtmjb" pod "f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4" (UID: "f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4"). InnerVolumeSpecName "kube-api-access-rtmjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:10.315722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.315631    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:10.315722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.315671    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmjb\" (UniqueName: \"kubernetes.io/projected/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4-kube-api-access-rtmjb\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:10.987028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.986998    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"
Nov 03 20:13:10.987238 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.987030    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz" event={"ID":"f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4","Type":"ContainerDied","Data":"9e97f8629668ddf0a440bdb506c7dc1a737cc3545473d251944670f479035222"}
Nov 03 20:13:10.987238 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:10.987088    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e97f8629668ddf0a440bdb506c7dc1a737cc3545473d251944670f479035222"
Nov 03 20:13:17.437263 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:17.437221    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:20.573139 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:13:20.573103    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:13:30.021802 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:30.021755    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"]
Nov 03 20:13:30.023949 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:30.023914    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2215c816-ptcqm"]
Nov 03 20:13:30.856585 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:30.856539    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9683b009-84b8-4509-b43b-1a18d6646f7c" path="/var/lib/kubelet/pods/9683b009-84b8-4509-b43b-1a18d6646f7c/volumes"
Nov 03 20:13:32.222114 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.222078    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"]
Nov 03 20:13:32.222567 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.222526    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4" containerName="revoke-keys"
Nov 03 20:13:32.222567 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.222538    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4" containerName="revoke-keys"
Nov 03 20:13:32.222643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.222611    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4" containerName="revoke-keys"
Nov 03 20:13:32.226113 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.226094    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:32.229393 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.229362    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:13:32.229556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.229514    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:13:32.229556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.229541    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:13:32.229700 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.229648    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:13:32.231872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.231844    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"]
Nov 03 20:13:32.311680 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.311641    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s52vb\" (UniqueName: \"kubernetes.io/projected/c4c5055f-eced-432e-bbed-620e7fdcad28-kube-api-access-s52vb\") pod \"maas-api-revoke-sub-e2fc9c5b-6r878\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:32.311680 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.311686    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6r878\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:32.311941 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.311712    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6r878\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:32.412768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.412731    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6r878\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:32.412768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.412772    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6r878\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:32.413004 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.412867    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s52vb\" (UniqueName: \"kubernetes.io/projected/c4c5055f-eced-432e-bbed-620e7fdcad28-kube-api-access-s52vb\") pod \"maas-api-revoke-sub-e2fc9c5b-6r878\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:32.413620 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.413599    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6r878\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:32.415272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.415254    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6r878\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:32.420412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.420381    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s52vb\" (UniqueName: \"kubernetes.io/projected/c4c5055f-eced-432e-bbed-620e7fdcad28-kube-api-access-s52vb\") pod \"maas-api-revoke-sub-e2fc9c5b-6r878\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:32.541529 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.541433    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:32.816662 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:32.816632    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"]
Nov 03 20:13:32.817862 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:13:32.817832    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c5055f_eced_432e_bbed_620e7fdcad28.slice/crio-1804700dab4eb629161f2ff4d1c267c6070bf589b27e334ee8ba2d28dca6ad99 WatchSource:0}: Error finding container 1804700dab4eb629161f2ff4d1c267c6070bf589b27e334ee8ba2d28dca6ad99: Status 404 returned error can't find the container with id 1804700dab4eb629161f2ff4d1c267c6070bf589b27e334ee8ba2d28dca6ad99
Nov 03 20:13:33.135519 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:33.135479    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878" event={"ID":"c4c5055f-eced-432e-bbed-620e7fdcad28","Type":"ContainerStarted","Data":"1804700dab4eb629161f2ff4d1c267c6070bf589b27e334ee8ba2d28dca6ad99"}
Nov 03 20:13:34.143113 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:34.143070    4531 generic.go:356] "Generic (PLEG): container finished" podID="c4c5055f-eced-432e-bbed-620e7fdcad28" containerID="b86bbe2de73132d3ada9b24d00f6a802ea72632d3587ced33333e8cac2738eab" exitCode=0
Nov 03 20:13:34.143554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:34.143154    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878" event={"ID":"c4c5055f-eced-432e-bbed-620e7fdcad28","Type":"ContainerDied","Data":"b86bbe2de73132d3ada9b24d00f6a802ea72632d3587ced33333e8cac2738eab"}
Nov 03 20:13:35.280375 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:35.280349    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:35.440945 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:35.440852    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-openshift-ca\") pod \"c4c5055f-eced-432e-bbed-620e7fdcad28\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") "
Nov 03 20:13:35.440945 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:35.440922    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-xks-ca\") pod \"c4c5055f-eced-432e-bbed-620e7fdcad28\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") "
Nov 03 20:13:35.441262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:35.440989    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4c5055f-eced-432e-bbed-620e7fdcad28-kube-api-access-s52vb\" (UniqueName: \"kubernetes.io/projected/c4c5055f-eced-432e-bbed-620e7fdcad28-kube-api-access-s52vb\") pod \"c4c5055f-eced-432e-bbed-620e7fdcad28\" (UID: \"c4c5055f-eced-432e-bbed-620e7fdcad28\") "
Nov 03 20:13:35.441342 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:35.441255    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-openshift-ca" pod "c4c5055f-eced-432e-bbed-620e7fdcad28" (UID: "c4c5055f-eced-432e-bbed-620e7fdcad28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:35.443108 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:35.443079    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-xks-ca" pod "c4c5055f-eced-432e-bbed-620e7fdcad28" (UID: "c4c5055f-eced-432e-bbed-620e7fdcad28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:13:35.443169 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:35.443144    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c5055f-eced-432e-bbed-620e7fdcad28-kube-api-access-s52vb" pod "c4c5055f-eced-432e-bbed-620e7fdcad28" (UID: "c4c5055f-eced-432e-bbed-620e7fdcad28"). InnerVolumeSpecName "kube-api-access-s52vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:35.542690 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:35.542648    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:35.542690 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:35.542683    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s52vb\" (UniqueName: \"kubernetes.io/projected/c4c5055f-eced-432e-bbed-620e7fdcad28-kube-api-access-s52vb\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:35.542690 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:35.542695    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c5055f-eced-432e-bbed-620e7fdcad28-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:36.155677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:36.155650    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"
Nov 03 20:13:36.155873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:36.155681    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878" event={"ID":"c4c5055f-eced-432e-bbed-620e7fdcad28","Type":"ContainerDied","Data":"1804700dab4eb629161f2ff4d1c267c6070bf589b27e334ee8ba2d28dca6ad99"}
Nov 03 20:13:36.155873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:36.155731    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1804700dab4eb629161f2ff4d1c267c6070bf589b27e334ee8ba2d28dca6ad99"
Nov 03 20:13:36.382019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:36.381982    4531 scope.go:122] "RemoveContainer" containerID="77c000d17f4c492a6ee96b1f6a2bfcd3fb79512e91b30226d5e588a9226df233"
Nov 03 20:13:36.391523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:36.391499    4531 scope.go:122] "RemoveContainer" containerID="e6df4de8909aecc1169ebd66717384f15ffd66831abb09c6943f2dacb65683be"
Nov 03 20:13:36.403319 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:36.403293    4531 scope.go:122] "RemoveContainer" containerID="805eae836ce462caff3d5bac2d9bce3477ac3c29cffb470f552d80e1dea14551"
Nov 03 20:13:36.412489 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:36.412468    4531 scope.go:122] "RemoveContainer" containerID="ec75c5deee30f05491dddad9caca6eddcee342a61d1ff9f3d7e78aa695b3769f"
Nov 03 20:13:36.420726 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:36.420705    4531 scope.go:122] "RemoveContainer" containerID="aaac5e997d25e9cbdb781236ba6f40f412238b80f5efcf9d28f212808901b12b"
Nov 03 20:13:36.428387 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:36.428356    4531 scope.go:122] "RemoveContainer" containerID="813b047ecb68b9b8f94fbe09e90e5985a2a2cfc35b117ef711abcaaf41512c00"
Nov 03 20:13:42.441225 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.441151    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"]
Nov 03 20:13:42.441902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.441824    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c4c5055f-eced-432e-bbed-620e7fdcad28" containerName="revoke-keys"
Nov 03 20:13:42.441902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.441846    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4c5055f-eced-432e-bbed-620e7fdcad28" containerName="revoke-keys"
Nov 03 20:13:42.442042 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.441966    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4c5055f-eced-432e-bbed-620e7fdcad28" containerName="revoke-keys"
Nov 03 20:13:42.447300 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.447273    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:42.451458 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.451433    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:13:42.451669 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.451451    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:13:42.451860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.451477    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"]
Nov 03 20:13:42.451860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.451765    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:13:42.451986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.451892    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:13:42.609402 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.609364    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-jctzd\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:42.609596 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.609416    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nrht\" (UniqueName: \"kubernetes.io/projected/2882008e-047a-41f5-8a1e-35490cb73074-kube-api-access-7nrht\") pod \"maas-api-revoke-sub-5a29efb3-jctzd\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:42.609643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.609602    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-jctzd\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:42.710250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.710146    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-jctzd\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:42.710250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.710230    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nrht\" (UniqueName: \"kubernetes.io/projected/2882008e-047a-41f5-8a1e-35490cb73074-kube-api-access-7nrht\") pod \"maas-api-revoke-sub-5a29efb3-jctzd\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:42.710509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.710346    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-jctzd\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:42.710981 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.710952    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-jctzd\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:42.712631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.712608    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-jctzd\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:42.717474 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.717451    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nrht\" (UniqueName: \"kubernetes.io/projected/2882008e-047a-41f5-8a1e-35490cb73074-kube-api-access-7nrht\") pod \"maas-api-revoke-sub-5a29efb3-jctzd\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:42.764613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:42.764577    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:43.036260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:43.036234    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"]
Nov 03 20:13:43.038286 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:13:43.038249    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2882008e_047a_41f5_8a1e_35490cb73074.slice/crio-3eecbe2657983a9b0bcc7d82ea2b2975c08d6f04ef5810288675848bd5ab16b8 WatchSource:0}: Error finding container 3eecbe2657983a9b0bcc7d82ea2b2975c08d6f04ef5810288675848bd5ab16b8: Status 404 returned error can't find the container with id 3eecbe2657983a9b0bcc7d82ea2b2975c08d6f04ef5810288675848bd5ab16b8
Nov 03 20:13:43.207095 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:43.207051    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd" event={"ID":"2882008e-047a-41f5-8a1e-35490cb73074","Type":"ContainerStarted","Data":"3eecbe2657983a9b0bcc7d82ea2b2975c08d6f04ef5810288675848bd5ab16b8"}
Nov 03 20:13:44.029122 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:44.029055    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"]
Nov 03 20:13:44.032802 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:44.032749    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a731ee-xs65l"]
Nov 03 20:13:44.219597 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:44.219491    4531 generic.go:356] "Generic (PLEG): container finished" podID="2882008e-047a-41f5-8a1e-35490cb73074" containerID="397ff534a280be06ead3bdf74025d72d13b16785f919fb26f0d576717f6b86c6" exitCode=0
Nov 03 20:13:44.219803 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:44.219628    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd" event={"ID":"2882008e-047a-41f5-8a1e-35490cb73074","Type":"ContainerDied","Data":"397ff534a280be06ead3bdf74025d72d13b16785f919fb26f0d576717f6b86c6"}
Nov 03 20:13:44.877198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:44.876334    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc749a2a-2ef9-4f38-b842-4fc8bb058929" path="/var/lib/kubelet/pods/fc749a2a-2ef9-4f38-b842-4fc8bb058929/volumes"
Nov 03 20:13:45.988073 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:45.988036    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:46.129198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.128742    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-xks-ca\") pod \"2882008e-047a-41f5-8a1e-35490cb73074\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") "
Nov 03 20:13:46.129198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.128931    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2882008e-047a-41f5-8a1e-35490cb73074-kube-api-access-7nrht\" (UniqueName: \"kubernetes.io/projected/2882008e-047a-41f5-8a1e-35490cb73074-kube-api-access-7nrht\") pod \"2882008e-047a-41f5-8a1e-35490cb73074\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") "
Nov 03 20:13:46.129198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.128995    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-openshift-ca\") pod \"2882008e-047a-41f5-8a1e-35490cb73074\" (UID: \"2882008e-047a-41f5-8a1e-35490cb73074\") "
Nov 03 20:13:46.131204 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.130343    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-openshift-ca" pod "2882008e-047a-41f5-8a1e-35490cb73074" (UID: "2882008e-047a-41f5-8a1e-35490cb73074"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:46.135192 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.132587    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2882008e-047a-41f5-8a1e-35490cb73074-kube-api-access-7nrht" pod "2882008e-047a-41f5-8a1e-35490cb73074" (UID: "2882008e-047a-41f5-8a1e-35490cb73074"). InnerVolumeSpecName "kube-api-access-7nrht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:46.139193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.135529    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-xks-ca" pod "2882008e-047a-41f5-8a1e-35490cb73074" (UID: "2882008e-047a-41f5-8a1e-35490cb73074"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:13:46.232898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.232850    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:46.233439 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.233413    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nrht\" (UniqueName: \"kubernetes.io/projected/2882008e-047a-41f5-8a1e-35490cb73074-kube-api-access-7nrht\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:46.233593 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.233578    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2882008e-047a-41f5-8a1e-35490cb73074-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:46.244332 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.244305    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"
Nov 03 20:13:46.244499 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.244342    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd" event={"ID":"2882008e-047a-41f5-8a1e-35490cb73074","Type":"ContainerDied","Data":"3eecbe2657983a9b0bcc7d82ea2b2975c08d6f04ef5810288675848bd5ab16b8"}
Nov 03 20:13:46.244499 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:46.244441    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eecbe2657983a9b0bcc7d82ea2b2975c08d6f04ef5810288675848bd5ab16b8"
Nov 03 20:13:50.572725 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:13:50.572682    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:13:57.779829 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.779777    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"]
Nov 03 20:13:57.782326 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.782287    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9tttz"]
Nov 03 20:13:57.809487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.809438    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"]
Nov 03 20:13:57.809873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.809854    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2882008e-047a-41f5-8a1e-35490cb73074" containerName="revoke-keys"
Nov 03 20:13:57.809873 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.809868    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2882008e-047a-41f5-8a1e-35490cb73074" containerName="revoke-keys"
Nov 03 20:13:57.810111 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.809945    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2882008e-047a-41f5-8a1e-35490cb73074" containerName="revoke-keys"
Nov 03 20:13:57.813230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.813206    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:13:57.817561 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.817447    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:13:57.817561 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.817533    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:13:57.817989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.817965    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:13:57.818354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.818329    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:13:57.824052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.823976    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"]
Nov 03 20:13:57.845294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.845258    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4qlr\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:13:57.845498 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.845304    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6qj\" (UniqueName: \"kubernetes.io/projected/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-kube-api-access-7h6qj\") pod \"maas-api-revoke-sub-3d4156f1-d4qlr\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:13:57.845575 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.845528    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4qlr\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:13:57.947035 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.946993    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4qlr\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:13:57.947248 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.947125    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4qlr\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:13:57.947323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.947196    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6qj\" (UniqueName: \"kubernetes.io/projected/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-kube-api-access-7h6qj\") pod \"maas-api-revoke-sub-3d4156f1-d4qlr\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:13:57.947834 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.947815    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4qlr\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:13:57.949447 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.949429    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4qlr\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:13:57.954839 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:57.954813    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6qj\" (UniqueName: \"kubernetes.io/projected/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-kube-api-access-7h6qj\") pod \"maas-api-revoke-sub-3d4156f1-d4qlr\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:13:58.129479 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:58.129452    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:13:58.404294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:58.404262    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"]
Nov 03 20:13:58.405392 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:13:58.405365    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4123ea7d_bdbf_4d51_ac79_e9612073a5c9.slice/crio-1b9a5539b013641779c7ac4f0c8be41653ce472858cfb82a023f658751690730 WatchSource:0}: Error finding container 1b9a5539b013641779c7ac4f0c8be41653ce472858cfb82a023f658751690730: Status 404 returned error can't find the container with id 1b9a5539b013641779c7ac4f0c8be41653ce472858cfb82a023f658751690730
Nov 03 20:13:58.857218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:58.857115    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4" path="/var/lib/kubelet/pods/f8367c5e-a1b6-476b-9f31-1a1fc1b29bd4/volumes"
Nov 03 20:13:59.313854 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:59.313823    4531 generic.go:356] "Generic (PLEG): container finished" podID="4123ea7d-bdbf-4d51-ac79-e9612073a5c9" containerID="84d8b1b03d527c28d0a98d801677921ddf4e176e379099624488de3c0a74fef8" exitCode=0
Nov 03 20:13:59.314041 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:59.313868    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr" event={"ID":"4123ea7d-bdbf-4d51-ac79-e9612073a5c9","Type":"ContainerDied","Data":"84d8b1b03d527c28d0a98d801677921ddf4e176e379099624488de3c0a74fef8"}
Nov 03 20:13:59.314041 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:13:59.313906    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr" event={"ID":"4123ea7d-bdbf-4d51-ac79-e9612073a5c9","Type":"ContainerStarted","Data":"1b9a5539b013641779c7ac4f0c8be41653ce472858cfb82a023f658751690730"}
Nov 03 20:14:00.453242 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:00.453215    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:14:00.571764 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:00.571666    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-xks-ca\") pod \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") "
Nov 03 20:14:00.571764 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:00.571716    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-openshift-ca\") pod \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") "
Nov 03 20:14:00.571764 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:00.571758    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-kube-api-access-7h6qj\" (UniqueName: \"kubernetes.io/projected/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-kube-api-access-7h6qj\") pod \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\" (UID: \"4123ea7d-bdbf-4d51-ac79-e9612073a5c9\") "
Nov 03 20:14:00.572206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:00.572151    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-openshift-ca" pod "4123ea7d-bdbf-4d51-ac79-e9612073a5c9" (UID: "4123ea7d-bdbf-4d51-ac79-e9612073a5c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:14:00.573879 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:00.573857    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-xks-ca" pod "4123ea7d-bdbf-4d51-ac79-e9612073a5c9" (UID: "4123ea7d-bdbf-4d51-ac79-e9612073a5c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:14:00.573948 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:00.573930    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-kube-api-access-7h6qj" pod "4123ea7d-bdbf-4d51-ac79-e9612073a5c9" (UID: "4123ea7d-bdbf-4d51-ac79-e9612073a5c9"). InnerVolumeSpecName "kube-api-access-7h6qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:00.672743 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:00.672701    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:00.672743 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:00.672744    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:00.672970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:00.672755    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h6qj\" (UniqueName: \"kubernetes.io/projected/4123ea7d-bdbf-4d51-ac79-e9612073a5c9-kube-api-access-7h6qj\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:01.324851 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:01.324820    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"
Nov 03 20:14:01.325040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:01.324858    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr" event={"ID":"4123ea7d-bdbf-4d51-ac79-e9612073a5c9","Type":"ContainerDied","Data":"1b9a5539b013641779c7ac4f0c8be41653ce472858cfb82a023f658751690730"}
Nov 03 20:14:01.325040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:01.324904    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b9a5539b013641779c7ac4f0c8be41653ce472858cfb82a023f658751690730"
Nov 03 20:14:12.869398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:12.869307    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:13.196630 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.196594    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"]
Nov 03 20:14:13.197079 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.197064    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4123ea7d-bdbf-4d51-ac79-e9612073a5c9" containerName="revoke-keys"
Nov 03 20:14:13.197136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.197081    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4123ea7d-bdbf-4d51-ac79-e9612073a5c9" containerName="revoke-keys"
Nov 03 20:14:13.197221 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.197150    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4123ea7d-bdbf-4d51-ac79-e9612073a5c9" containerName="revoke-keys"
Nov 03 20:14:13.201002 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.200979    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.208816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.208791    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-030599-openshift-default-dockercfg-tgfxd\""
Nov 03 20:14:13.217364 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.217334    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"]
Nov 03 20:14:13.305521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.305482    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsljh\" (UniqueName: \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-kube-api-access-dsljh\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.305721 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.305527    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-certs\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.305721 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.305614    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-token\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.305721 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.305712    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-data\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.305842 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.305733    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-socket\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.305842 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.305787    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-credential-socket\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.305913 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.305844    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-podinfo\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.305913 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.305881    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-envoy\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.306019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.305991    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6cfe10-1e23-484c-a481-91f2daba30a4-istiod-ca-cert\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.309523 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.309477    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:13.406931 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.406898    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-envoy\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.406939    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6cfe10-1e23-484c-a481-91f2daba30a4-istiod-ca-cert\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.406972    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsljh\" (UniqueName: \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-kube-api-access-dsljh\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.406989    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-certs\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.407099    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-token\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.407254    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-data\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.407287    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-socket\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.407317    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-credential-socket\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.407385    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-podinfo\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.407521    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-certs\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.407643    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-data\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.407721    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-credential-socket\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.407834    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6cfe10-1e23-484c-a481-91f2daba30a4-istiod-ca-cert\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.407915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.407835    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-socket\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.409484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.409449    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-envoy\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.409883 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.409866    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-podinfo\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.415473 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.415449    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsljh\" (UniqueName: \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-kube-api-access-dsljh\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.416327 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.416306    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-token\") pod \"e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") " pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.455451 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.455349    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:13.517412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.517376    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:13.799482 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.799448    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"]
Nov 03 20:14:13.800406 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:14:13.800379    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6cfe10_1e23_484c_a481_91f2daba30a4.slice/crio-d0c139c65eafbd56835c6ba664bae5912498ad74aceb9097900a9322bd348097 WatchSource:0}: Error finding container d0c139c65eafbd56835c6ba664bae5912498ad74aceb9097900a9322bd348097: Status 404 returned error can't find the container with id d0c139c65eafbd56835c6ba664bae5912498ad74aceb9097900a9322bd348097
Nov 03 20:14:13.802833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.802794    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:13.802917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.802884    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:13.802955 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:13.802924    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:14.402651 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:14.402610    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" event={"ID":"5d6cfe10-1e23-484c-a481-91f2daba30a4","Type":"ContainerStarted","Data":"145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc"}
Nov 03 20:14:14.403078 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:14.402670    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" event={"ID":"5d6cfe10-1e23-484c-a481-91f2daba30a4","Type":"ContainerStarted","Data":"d0c139c65eafbd56835c6ba664bae5912498ad74aceb9097900a9322bd348097"}
Nov 03 20:14:14.422841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:14.422774    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" podStartSLOduration=1.422756702 podStartE2EDuration="1.422756702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:14.422243058 +0000 UTC m=+2567.692366353" watchObservedRunningTime="2026-11-03 20:14:14.422756702 +0000 UTC m=+2567.692879997"
Nov 03 20:14:14.517814 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:14.517777    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:14.523087 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:14.523057    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:15.408303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:15.408275    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:15.409371 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:15.409345    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:16.702474 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:16.702430    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:16.874472 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:16.874437    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:17.506136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.506100    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:17.834607 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.834526    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"]
Nov 03 20:14:17.838640 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.838612    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:17.844289 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.844260    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-030599-serving-cert\""
Nov 03 20:14:17.849243 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.848763    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:14:17.849243 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.848854    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:14:17.851077 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.851050    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"]
Nov 03 20:14:17.881341 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.881308    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rj6j4\""
Nov 03 20:14:17.881530 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.881380    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:14:17.951835 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.951796    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:17.952027 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.951859    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:17.952100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.952031    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98c2a76-4578-4a75-b88c-d66067ea3cc4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:17.952100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.952075    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl7wg\" (UniqueName: \"kubernetes.io/projected/e98c2a76-4578-4a75-b88c-d66067ea3cc4-kube-api-access-cl7wg\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:17.979124 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.979095    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:17.989849 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.989817    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"]
Nov 03 20:14:17.993703 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:17.993681    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:14:18.009381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.009353    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8bbf8aed-dockercfg-x8bxp\""
Nov 03 20:14:18.009573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.009541    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-030599\""
Nov 03 20:14:18.013363 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.013340    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"]
Nov 03 20:14:18.053238 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.053202    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98c2a76-4578-4a75-b88c-d66067ea3cc4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:18.053427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.053264    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl7wg\" (UniqueName: \"kubernetes.io/projected/e98c2a76-4578-4a75-b88c-d66067ea3cc4-kube-api-access-cl7wg\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:18.053427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.053327    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:18.053427 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.053371    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:18.054240 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.054035    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98c2a76-4578-4a75-b88c-d66067ea3cc4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:18.056272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.056250    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:18.057101 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.057079    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:18.057720 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.057700    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"]
Nov 03 20:14:18.061422 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.061403    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:14:18.072052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.072025    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl7wg\" (UniqueName: \"kubernetes.io/projected/e98c2a76-4578-4a75-b88c-d66067ea3cc4-kube-api-access-cl7wg\") pod \"maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:18.075049 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.075016    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"]
Nov 03 20:14:18.154640 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.154593    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c3a53b-edaf-447c-948f-d498a3dd966b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8\" (UID: \"48c3a53b-edaf-447c-948f-d498a3dd966b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:14:18.154876 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.154657    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjln\" (UniqueName: \"kubernetes.io/projected/48c3a53b-edaf-447c-948f-d498a3dd966b-kube-api-access-kpjln\") pod \"payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8\" (UID: \"48c3a53b-edaf-447c-948f-d498a3dd966b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:14:18.154876 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.154770    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz\" (UID: \"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:14:18.155065 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.154933    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ktf7\" (UniqueName: \"kubernetes.io/projected/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-kube-api-access-2ktf7\") pod \"payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz\" (UID: \"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:14:18.161416 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.161233    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:18.161960 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.161936    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:18.255975 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.255887    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz\" (UID: \"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:14:18.256188 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.256008    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ktf7\" (UniqueName: \"kubernetes.io/projected/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-kube-api-access-2ktf7\") pod \"payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz\" (UID: \"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:14:18.256188 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.256058    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c3a53b-edaf-447c-948f-d498a3dd966b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8\" (UID: \"48c3a53b-edaf-447c-948f-d498a3dd966b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:14:18.256188 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.256078    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjln\" (UniqueName: \"kubernetes.io/projected/48c3a53b-edaf-447c-948f-d498a3dd966b-kube-api-access-kpjln\") pod \"payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8\" (UID: \"48c3a53b-edaf-447c-948f-d498a3dd966b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:14:18.257286 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.257211    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz\" (UID: \"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:14:18.258338 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.258316    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c3a53b-edaf-447c-948f-d498a3dd966b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8\" (UID: \"48c3a53b-edaf-447c-948f-d498a3dd966b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:14:18.265136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.265113    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjln\" (UniqueName: \"kubernetes.io/projected/48c3a53b-edaf-447c-948f-d498a3dd966b-kube-api-access-kpjln\") pod \"payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8\" (UID: \"48c3a53b-edaf-447c-948f-d498a3dd966b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:14:18.265395 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.265379    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ktf7\" (UniqueName: \"kubernetes.io/projected/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-kube-api-access-2ktf7\") pod \"payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz\" (UID: \"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:14:18.307093 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.307059    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:14:18.320807 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.320616    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:18.392960 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.392920    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:14:18.449035 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.449007    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"]
Nov 03 20:14:18.450404 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:14:18.450358    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98c2a76_4578_4a75_b88c_d66067ea3cc4.slice/crio-8e04eaad417c85ad2c470d4a0dfd91cc73cc6d8949903d2bc7a20a2d47429615 WatchSource:0}: Error finding container 8e04eaad417c85ad2c470d4a0dfd91cc73cc6d8949903d2bc7a20a2d47429615: Status 404 returned error can't find the container with id 8e04eaad417c85ad2c470d4a0dfd91cc73cc6d8949903d2bc7a20a2d47429615
Nov 03 20:14:18.454018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.453994    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:14:18.582955 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.582929    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"]
Nov 03 20:14:18.584967 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:14:18.584936    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c3a53b_edaf_447c_948f_d498a3dd966b.slice/crio-c24d02f8484a11f565b44187ee528453e57ae1ecd0ea9d6b3004efb0fbf0caf2 WatchSource:0}: Error finding container c24d02f8484a11f565b44187ee528453e57ae1ecd0ea9d6b3004efb0fbf0caf2: Status 404 returned error can't find the container with id c24d02f8484a11f565b44187ee528453e57ae1ecd0ea9d6b3004efb0fbf0caf2
Nov 03 20:14:18.685579 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:18.685550    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"]
Nov 03 20:14:18.687407 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:14:18.687366    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4bc543_ceea_49fc_a3e7_68ad7dcefb6d.slice/crio-5434a49da7fa8a901643ca0c3f85be4af774c5311de6ef52b68b004cc2ac82c4 WatchSource:0}: Error finding container 5434a49da7fa8a901643ca0c3f85be4af774c5311de6ef52b68b004cc2ac82c4: Status 404 returned error can't find the container with id 5434a49da7fa8a901643ca0c3f85be4af774c5311de6ef52b68b004cc2ac82c4
Nov 03 20:14:19.433531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.433491    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz" event={"ID":"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d","Type":"ContainerStarted","Data":"86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e"}
Nov 03 20:14:19.433984 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.433545    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz" event={"ID":"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d","Type":"ContainerStarted","Data":"5434a49da7fa8a901643ca0c3f85be4af774c5311de6ef52b68b004cc2ac82c4"}
Nov 03 20:14:19.433984 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.433679    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:14:19.435271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.435244    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8" event={"ID":"48c3a53b-edaf-447c-948f-d498a3dd966b","Type":"ContainerStarted","Data":"ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb"}
Nov 03 20:14:19.435381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.435298    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:14:19.435381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.435320    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8" event={"ID":"48c3a53b-edaf-447c-948f-d498a3dd966b","Type":"ContainerStarted","Data":"c24d02f8484a11f565b44187ee528453e57ae1ecd0ea9d6b3004efb0fbf0caf2"}
Nov 03 20:14:19.436939 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.436909    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c" event={"ID":"e98c2a76-4578-4a75-b88c-d66067ea3cc4","Type":"ContainerStarted","Data":"ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea"}
Nov 03 20:14:19.437048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.436964    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c" event={"ID":"e98c2a76-4578-4a75-b88c-d66067ea3cc4","Type":"ContainerStarted","Data":"8e04eaad417c85ad2c470d4a0dfd91cc73cc6d8949903d2bc7a20a2d47429615"}
Nov 03 20:14:19.437048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.436993    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:19.452586 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.452537    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz" podStartSLOduration=1.4525208 podStartE2EDuration="1.4525208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:19.450143686 +0000 UTC m=+2572.720266980" watchObservedRunningTime="2026-11-03 20:14:19.4525208 +0000 UTC m=+2572.722644095"
Nov 03 20:14:19.490034 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.489974    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c" podStartSLOduration=2.114712831 podStartE2EDuration="2.489961189s" totalImagesPullingTime="375.248358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:19.466240456 +0000 UTC m=+2572.736363750" watchObservedRunningTime="2026-11-03 20:14:19.489961189 +0000 UTC m=+2572.760084512"
Nov 03 20:14:19.490283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.490094    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8" podStartSLOduration=2.490087743 podStartE2EDuration="2.490087743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:19.487511096 +0000 UTC m=+2572.757634392" watchObservedRunningTime="2026-11-03 20:14:19.490087743 +0000 UTC m=+2572.760211039"
Nov 03 20:14:19.979349 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:19.979315    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:20.118912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:20.118804    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:20.534785 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:20.534737    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:20.573447 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:14:20.573404    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:14:21.561921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:21.561876    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:22.250769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:22.250728    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:23.169860 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:23.169816    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:24.003876 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:24.003804    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:24.959311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:24.959267    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:25.465685 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:25.465647    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:14:25.747260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:25.744699    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:26.557497 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:26.557465    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:27.393236 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:27.393196    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:28.028882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:28.028849    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"]
Nov 03 20:14:28.032879 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:28.032846    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-2hgfz"]
Nov 03 20:14:28.157991 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:28.157960    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:28.858646 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:28.858607    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87058489-0b28-480e-a0a7-80f1fdb72817" path="/var/lib/kubelet/pods/87058489-0b28-480e-a0a7-80f1fdb72817/volumes"
Nov 03 20:14:28.884088 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:28.884051    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:29.628630 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:29.628590    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:30.409636 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:30.409601    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:30.409893 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:30.409659    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 03 20:14:30.419783 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:30.419722    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:30.454934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:30.454900    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:14:30.455135 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:30.454973    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:14:30.551796 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:30.551755    4531 generic.go:356] "Generic (PLEG): container finished" podID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerID="145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc" exitCode=0
Nov 03 20:14:30.551796 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:30.551799    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" event={"ID":"5d6cfe10-1e23-484c-a481-91f2daba30a4","Type":"ContainerDied","Data":"145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc"}
Nov 03 20:14:30.552283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:30.552264    4531 scope.go:122] "RemoveContainer" containerID="145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc"
Nov 03 20:14:30.553801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:30.553768    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:30.553888 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:30.553836    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:30.553888 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:30.553870    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:31.171018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:31.170957    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:31.564879 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:31.564793    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" event={"ID":"5d6cfe10-1e23-484c-a481-91f2daba30a4","Type":"ContainerStarted","Data":"93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e"}
Nov 03 20:14:31.673462 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:31.673428    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:31.952198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:31.948073    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:32.261260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:32.258243    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:32.392562 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:32.392503    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:32.517865 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:32.517770    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:32.518135 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:32.518112    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:32.518203 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:32.518163    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 03 20:14:32.926358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:32.926301    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:33.518812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:33.518773    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:33.519285 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:33.518838    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 03 20:14:33.746856 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:33.746823    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:34.226243 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.226199    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"]
Nov 03 20:14:34.230315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.230283    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.234456 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.234428    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-030599\"/\"openshift-service-ca.crt\""
Nov 03 20:14:34.262627 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.262594    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-030599\"/\"default-dockercfg-hr8t8\""
Nov 03 20:14:34.262842 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.262602    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-030599\"/\"kube-root-ca.crt\""
Nov 03 20:14:34.263054 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.262684    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-030599\"/\"simulator-single-main-030599-kserve-self-signed-certs\""
Nov 03 20:14:34.269204 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.269159    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"]
Nov 03 20:14:34.290224 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.290191    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:34.377303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.376798    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-model-cache\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.377303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.376854    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tls-certs\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.377303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.376886    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-dshm\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.377303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.377014    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tmp-dir\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.377303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.377071    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-home\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.377303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.377119    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9n7q\" (UniqueName: \"kubernetes.io/projected/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-kube-api-access-f9n7q\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.477871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.477765    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-dshm\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.477871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.477866    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tmp-dir\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.478143 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.477897    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-home\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.478143 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.477927    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9n7q\" (UniqueName: \"kubernetes.io/projected/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-kube-api-access-f9n7q\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.478143 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.477962    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-model-cache\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.478143 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.477991    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tls-certs\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.478426 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.478400    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-home\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.478485 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.478431    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tmp-dir\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.478580 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.478503    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-model-cache\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.480464 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.480441    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-dshm\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.480577 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.480464    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tls-certs\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.486109 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.486086    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9n7q\" (UniqueName: \"kubernetes.io/projected/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-kube-api-access-f9n7q\") pod \"simulator-single-main-030599-kserve-5c6f6c947d-wxmnn\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") " pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.518211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.518161    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:34.518367 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.518240    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 03 20:14:34.573153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.573110    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:34.669429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.668675    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:34.888870 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:34.888841    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"]
Nov 03 20:14:34.890849 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:14:34.890807    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cf3b7d1_66f0_4a2a_914a_0d459285aa72.slice/crio-53e849a9f167ed64cbaaebd13ef5663cf6fd38cc745b3fafb0936151847145b3 WatchSource:0}: Error finding container 53e849a9f167ed64cbaaebd13ef5663cf6fd38cc745b3fafb0936151847145b3: Status 404 returned error can't find the container with id 53e849a9f167ed64cbaaebd13ef5663cf6fd38cc745b3fafb0936151847145b3
Nov 03 20:14:35.168691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:35.168598    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:35.515517 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:35.515435    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:35.517885 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:35.517861    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:35.518004 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:35.517916    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 03 20:14:35.601885 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:35.601842    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn" event={"ID":"9cf3b7d1-66f0-4a2a-914a-0d459285aa72","Type":"ContainerStarted","Data":"b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302"}
Nov 03 20:14:35.602354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:35.601902    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn" event={"ID":"9cf3b7d1-66f0-4a2a-914a-0d459285aa72","Type":"ContainerStarted","Data":"53e849a9f167ed64cbaaebd13ef5663cf6fd38cc745b3fafb0936151847145b3"}
Nov 03 20:14:35.619668 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:35.619605    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn" podStartSLOduration=1.6195910150000001 podStartE2EDuration="1.619591015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:35.618430386 +0000 UTC m=+2588.888553681" watchObservedRunningTime="2026-11-03 20:14:35.619591015 +0000 UTC m=+2588.889714310"
Nov 03 20:14:36.442539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:36.442506    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:36.470222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:36.470190    4531 scope.go:122] "RemoveContainer" containerID="48e5ce457e0d9a86ab0862649d3283376360d609593c5104b9066bff8f0fde8f"
Nov 03 20:14:36.482290 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:36.482268    4531 scope.go:122] "RemoveContainer" containerID="5a427733517b60daff8a37d6eb5438f0606cf74853cde9e4d380ad30de2c89c5"
Nov 03 20:14:36.491401 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:36.491378    4531 scope.go:122] "RemoveContainer" containerID="31ac6e22e768809912b6142281f055d9f8e488123bc188f77a9e60bb81bd3c5c"
Nov 03 20:14:36.518158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:36.518132    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:36.518268 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:36.518210    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 03 20:14:36.574114 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:36.574076    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:37.269199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:37.269152    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:37.522754 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:37.522674    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:37.523200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:37.523152    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:37.523976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:37.523952    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:14:38.039658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:38.039600    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:38.818748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:38.818707    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:39.627062 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:39.627024    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:40.318272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:40.318237    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:41.012433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:41.012375    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:41.724283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:41.724251    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:42.425715 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:42.425680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:43.136676 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:43.136643    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:43.936422 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:43.936388    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:44.573444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:44.573409    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:44.586861 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:44.586832    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:44.670655 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:44.670625    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:14:44.779799 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:44.779770    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:45.661887 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:45.661761    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:46.709907 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:46.709874    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:47.577218 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:47.577181    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:48.528494 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:48.528454    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:49.355411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:49.355374    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:50.311190 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:50.311143    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:50.574050 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:14:50.573952    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:14:51.136855 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:51.136802    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:52.176840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:52.176804    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:52.946047 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:52.945993    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:53.724193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:53.724140    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:54.529517 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:54.529483    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:55.315867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:55.315825    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:56.019917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:56.019875    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:56.727467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:56.727430    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:57.530262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:57.530227    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:58.266923 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:58.266891    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:58.934709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:58.934673    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:59.630901 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:14:59.630869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:00.133462 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.133418    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"]
Nov 03 20:15:00.137433 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.137408    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"
Nov 03 20:15:00.148632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.146825    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:15:00.151533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.151490    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"]
Nov 03 20:15:00.256907 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.256864    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2mjh\" (UniqueName: \"kubernetes.io/projected/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-kube-api-access-v2mjh\") pod \"maas-api-key-cleanup-29895615-tndhb\" (UID: \"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"
Nov 03 20:15:00.257110 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.256920    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-tndhb\" (UID: \"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"
Nov 03 20:15:00.341265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.341235    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:00.358445 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.358410    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mjh\" (UniqueName: \"kubernetes.io/projected/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-kube-api-access-v2mjh\") pod \"maas-api-key-cleanup-29895615-tndhb\" (UID: \"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"
Nov 03 20:15:00.358628 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.358464    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-tndhb\" (UID: \"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"
Nov 03 20:15:00.359109 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.359091    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-tndhb\" (UID: \"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"
Nov 03 20:15:00.366016 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.365981    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2mjh\" (UniqueName: \"kubernetes.io/projected/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-kube-api-access-v2mjh\") pod \"maas-api-key-cleanup-29895615-tndhb\" (UID: \"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"
Nov 03 20:15:00.456455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.456373    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"
Nov 03 20:15:00.731750 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.731715    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"]
Nov 03 20:15:00.733243 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:15:00.733211    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b61be9_c5bf_4d0b_b8ba_d38d721a9457.slice/crio-067d7561b470bde05423e0e7c3fb87713ad3ae3e30c0cf1dc79b7f39283921af WatchSource:0}: Error finding container 067d7561b470bde05423e0e7c3fb87713ad3ae3e30c0cf1dc79b7f39283921af: Status 404 returned error can't find the container with id 067d7561b470bde05423e0e7c3fb87713ad3ae3e30c0cf1dc79b7f39283921af
Nov 03 20:15:00.753796 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:00.753766    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb" event={"ID":"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457","Type":"ContainerStarted","Data":"067d7561b470bde05423e0e7c3fb87713ad3ae3e30c0cf1dc79b7f39283921af"}
Nov 03 20:15:01.099026 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:01.098991    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:01.721689 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:01.721656    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:01.760847 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:01.760811    4531 generic.go:356] "Generic (PLEG): container finished" podID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerID="207f6ff5c62017ea23d73e8d85376212a108a698deab62fb677dc6cd774e245a" exitCode=60
Nov 03 20:15:01.761282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:01.760894    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb" event={"ID":"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457","Type":"ContainerDied","Data":"207f6ff5c62017ea23d73e8d85376212a108a698deab62fb677dc6cd774e245a"}
Nov 03 20:15:01.761282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:01.761098    4531 scope.go:122] "RemoveContainer" containerID="207f6ff5c62017ea23d73e8d85376212a108a698deab62fb677dc6cd774e245a"
Nov 03 20:15:02.423844 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:02.423780    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:02.769241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:02.769133    4531 generic.go:356] "Generic (PLEG): container finished" podID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerID="9e34f03d9064401c22d2b3b8671deeb470b871daa2fb6c3175398cc212a34387" exitCode=60
Nov 03 20:15:02.769241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:02.769221    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb" event={"ID":"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457","Type":"ContainerDied","Data":"9e34f03d9064401c22d2b3b8671deeb470b871daa2fb6c3175398cc212a34387"}
Nov 03 20:15:02.769745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:02.769278    4531 scope.go:122] "RemoveContainer" containerID="207f6ff5c62017ea23d73e8d85376212a108a698deab62fb677dc6cd774e245a"
Nov 03 20:15:02.769745 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:02.769532    4531 scope.go:122] "RemoveContainer" containerID="9e34f03d9064401c22d2b3b8671deeb470b871daa2fb6c3175398cc212a34387"
Nov 03 20:15:02.769859 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:15:02.769826    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-tndhb_odh-ai-gateway-infra(b1b61be9-c5bf-4d0b-b8ba-d38d721a9457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457"
Nov 03 20:15:03.205514 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:03.205483    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:03.786895 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:03.786856    4531 scope.go:122] "RemoveContainer" containerID="9e34f03d9064401c22d2b3b8671deeb470b871daa2fb6c3175398cc212a34387"
Nov 03 20:15:03.787338 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:15:03.787102    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-tndhb_odh-ai-gateway-infra(b1b61be9-c5bf-4d0b-b8ba-d38d721a9457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457"
Nov 03 20:15:04.132474 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:04.132434    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:04.831676 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:04.831635    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:05.534401 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:05.534362    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:06.295264 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:06.295232    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:07.019847 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:07.019814    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:07.724832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:07.724799    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:08.515675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:08.515611    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:09.223885 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:09.223836    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:09.939185 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:09.939129    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:10.635748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:10.635711    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:11.406944 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:11.406912    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:12.129133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:12.129099    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:12.836201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:12.836148    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:13.583769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:13.583733    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:14.250449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:14.250397    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:14.406687 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:14.406646    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:15.220197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:15.217652    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:15.851884 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:15.851852    4531 scope.go:122] "RemoveContainer" containerID="9e34f03d9064401c22d2b3b8671deeb470b871daa2fb6c3175398cc212a34387"
Nov 03 20:15:16.046300 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:16.046263    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:16.862213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:16.862159    4531 generic.go:356] "Generic (PLEG): container finished" podID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerID="ffa181db43fe81122247f9bf3f7435f3e294adb66d7debf384f57bee80dfa3f8" exitCode=60
Nov 03 20:15:16.862213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:16.862202    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb" event={"ID":"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457","Type":"ContainerDied","Data":"ffa181db43fe81122247f9bf3f7435f3e294adb66d7debf384f57bee80dfa3f8"}
Nov 03 20:15:16.862770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:16.862275    4531 scope.go:122] "RemoveContainer" containerID="9e34f03d9064401c22d2b3b8671deeb470b871daa2fb6c3175398cc212a34387"
Nov 03 20:15:16.862770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:16.862595    4531 scope.go:122] "RemoveContainer" containerID="ffa181db43fe81122247f9bf3f7435f3e294adb66d7debf384f57bee80dfa3f8"
Nov 03 20:15:16.862893 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:15:16.862873    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-tndhb_odh-ai-gateway-infra(b1b61be9-c5bf-4d0b-b8ba-d38d721a9457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457"
Nov 03 20:15:17.046896 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:17.046775    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:17.515111 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:17.515079    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:17.805729 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:17.805633    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:17.895262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:17.895222    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"]
Nov 03 20:15:18.032072 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.032046    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"
Nov 03 20:15:18.047160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.047127    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-maas-api-service-ca\") pod \"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457\" (UID: \"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457\") "
Nov 03 20:15:18.048485 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.047976    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-kube-api-access-v2mjh\" (UniqueName: \"kubernetes.io/projected/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-kube-api-access-v2mjh\") pod \"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457\" (UID: \"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457\") "
Nov 03 20:15:18.048641 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.048560    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-maas-api-service-ca" pod "b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" (UID: "b1b61be9-c5bf-4d0b-b8ba-d38d721a9457"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:18.050010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.049981    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-kube-api-access-v2mjh" pod "b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" (UID: "b1b61be9-c5bf-4d0b-b8ba-d38d721a9457"). InnerVolumeSpecName "kube-api-access-v2mjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:18.149573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.149534    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-maas-api-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:18.149573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.149567    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2mjh\" (UniqueName: \"kubernetes.io/projected/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457-kube-api-access-v2mjh\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:18.616725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.616687    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:18.875742 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.875675    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"
Nov 03 20:15:18.875742 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.875707    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb" event={"ID":"b1b61be9-c5bf-4d0b-b8ba-d38d721a9457","Type":"ContainerDied","Data":"067d7561b470bde05423e0e7c3fb87713ad3ae3e30c0cf1dc79b7f39283921af"}
Nov 03 20:15:18.875983 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.875767    4531 scope.go:122] "RemoveContainer" containerID="ffa181db43fe81122247f9bf3f7435f3e294adb66d7debf384f57bee80dfa3f8"
Nov 03 20:15:18.890556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.890529    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"]
Nov 03 20:15:18.892882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:18.892856    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-tndhb"]
Nov 03 20:15:19.156187 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:19.156149    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"]
Nov 03 20:15:19.273692 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:19.273656    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:19.403793 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:19.403759    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:20.117855 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:20.117760    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:20.573569 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:15:20.573530    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:15:20.803416 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:20.803380    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:20.857183 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:20.857074    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" path="/var/lib/kubelet/pods/b1b61be9-c5bf-4d0b-b8ba-d38d721a9457/volumes"
Nov 03 20:15:21.020329 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:21.020294    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"]
Nov 03 20:15:21.022546 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:21.022525    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-p9bgh"]
Nov 03 20:15:21.517580 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:21.517541    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:22.290660 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:22.290624    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:22.857266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:22.857223    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1f3cbd-31d0-4191-a720-33af8cb17560" path="/var/lib/kubelet/pods/df1f3cbd-31d0-4191-a720-33af8cb17560/volumes"
Nov 03 20:15:22.942045 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:22.941999    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:23.640854 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:23.640810    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:24.323263 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:24.323225    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:24.566600 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:24.566563    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"]
Nov 03 20:15:24.572154 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:24.572116    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"]
Nov 03 20:15:24.695425 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:24.695389    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:24.863684 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:24.863647    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:25.135777 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:25.135735    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:25.712753 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:25.712713    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"]
Nov 03 20:15:25.713156 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:25.712947    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c" podUID="e98c2a76-4578-4a75-b88c-d66067ea3cc4" containerName="maas-api" containerID="cri-o://ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea" gracePeriod=30
Nov 03 20:15:25.815836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:25.815782    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:25.970083 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:25.969990    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:26.107475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.107435    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:26.180607 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.180582    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:15:26.233971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.233883    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98c2a76-4578-4a75-b88c-d66067ea3cc4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98c2a76-4578-4a75-b88c-d66067ea3cc4-openshift-service-ca\") pod \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") "
Nov 03 20:15:26.233971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.233937    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-maas-api-tls\") pod \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") "
Nov 03 20:15:26.234298 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.234041    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98c2a76-4578-4a75-b88c-d66067ea3cc4-kube-api-access-cl7wg\" (UniqueName: \"kubernetes.io/projected/e98c2a76-4578-4a75-b88c-d66067ea3cc4-kube-api-access-cl7wg\") pod \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") "
Nov 03 20:15:26.234298 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.234197    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-metrics-certs\") pod \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\" (UID: \"e98c2a76-4578-4a75-b88c-d66067ea3cc4\") "
Nov 03 20:15:26.234459 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.234430    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98c2a76-4578-4a75-b88c-d66067ea3cc4-openshift-service-ca" pod "e98c2a76-4578-4a75-b88c-d66067ea3cc4" (UID: "e98c2a76-4578-4a75-b88c-d66067ea3cc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:26.234618 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.234595    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98c2a76-4578-4a75-b88c-d66067ea3cc4-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:26.236352 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.236317    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-maas-api-tls" pod "e98c2a76-4578-4a75-b88c-d66067ea3cc4" (UID: "e98c2a76-4578-4a75-b88c-d66067ea3cc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:15:26.236456 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.236351    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-metrics-certs" pod "e98c2a76-4578-4a75-b88c-d66067ea3cc4" (UID: "e98c2a76-4578-4a75-b88c-d66067ea3cc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:15:26.236456 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.236415    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98c2a76-4578-4a75-b88c-d66067ea3cc4-kube-api-access-cl7wg" pod "e98c2a76-4578-4a75-b88c-d66067ea3cc4" (UID: "e98c2a76-4578-4a75-b88c-d66067ea3cc4"). InnerVolumeSpecName "kube-api-access-cl7wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:26.335733 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.335696    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:26.335733 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.335728    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl7wg\" (UniqueName: \"kubernetes.io/projected/e98c2a76-4578-4a75-b88c-d66067ea3cc4-kube-api-access-cl7wg\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:26.335733 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.335739    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e98c2a76-4578-4a75-b88c-d66067ea3cc4-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:26.530762 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.530679    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:26.924422 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.924386    4531 generic.go:356] "Generic (PLEG): container finished" podID="e98c2a76-4578-4a75-b88c-d66067ea3cc4" containerID="ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea" exitCode=0
Nov 03 20:15:26.924893 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.924436    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"
Nov 03 20:15:26.924893 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.924467    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c" event={"ID":"e98c2a76-4578-4a75-b88c-d66067ea3cc4","Type":"ContainerDied","Data":"ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea"}
Nov 03 20:15:26.924893 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.924544    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c" event={"ID":"e98c2a76-4578-4a75-b88c-d66067ea3cc4","Type":"ContainerDied","Data":"8e04eaad417c85ad2c470d4a0dfd91cc73cc6d8949903d2bc7a20a2d47429615"}
Nov 03 20:15:26.924893 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.924573    4531 scope.go:122] "RemoveContainer" containerID="ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea"
Nov 03 20:15:26.934414 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.934385    4531 scope.go:122] "RemoveContainer" containerID="ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea"
Nov 03 20:15:26.934721 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:15:26.934692    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea\": container with ID starting with ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea not found: ID does not exist" containerID="ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea"
Nov 03 20:15:26.934795 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.934723    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea"} err="failed to get container status \"ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea\": rpc error: code = NotFound desc = could not find container \"ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea\": container with ID starting with ba9a88befe4044e061a866ef0bfc1ada257d0df81333e3a1d4bbce6606e339ea not found: ID does not exist"
Nov 03 20:15:26.947032 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.947001    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"]
Nov 03 20:15:26.949294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:26.949266    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-030599-6b666bc9c8-psg4c"]
Nov 03 20:15:28.857506 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:28.857470    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e98c2a76-4578-4a75-b88c-d66067ea3cc4" path="/var/lib/kubelet/pods/e98c2a76-4578-4a75-b88c-d66067ea3cc4/volumes"
Nov 03 20:15:29.567714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:29.567665    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz" podUID="dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d" containerName="payload-processing" containerID="cri-o://86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e" gracePeriod=25
Nov 03 20:15:29.573363 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:29.573333    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8" podUID="48c3a53b-edaf-447c-948f-d498a3dd966b" containerName="payload-pre-processing" containerID="cri-o://ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb" gracePeriod=25
Nov 03 20:15:30.446047 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:30.446001    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8" podUID="48c3a53b-edaf-447c-948f-d498a3dd966b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:15:30.446492 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:30.446011    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz" podUID="dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:15:32.310905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:32.310866    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:32.491556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:32.487785    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"]
Nov 03 20:15:32.492162 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:32.492117    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy" containerID="cri-o://93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e" gracePeriod=30
Nov 03 20:15:32.508693 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:32.508631    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:32.663181 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:32.663147    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:33.652060 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.652022    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"]
Nov 03 20:15:33.660549 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.660514    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4qlr"]
Nov 03 20:15:33.678970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.678937    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"]
Nov 03 20:15:33.679405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679389    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e98c2a76-4578-4a75-b88c-d66067ea3cc4" containerName="maas-api"
Nov 03 20:15:33.679467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679407    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98c2a76-4578-4a75-b88c-d66067ea3cc4" containerName="maas-api"
Nov 03 20:15:33.679467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679425    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerName="cleanup"
Nov 03 20:15:33.679467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679431    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerName="cleanup"
Nov 03 20:15:33.679467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679443    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerName="cleanup"
Nov 03 20:15:33.679467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679448    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerName="cleanup"
Nov 03 20:15:33.679467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679456    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerName="cleanup"
Nov 03 20:15:33.679467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679461    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerName="cleanup"
Nov 03 20:15:33.679700 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679555    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98c2a76-4578-4a75-b88c-d66067ea3cc4" containerName="maas-api"
Nov 03 20:15:33.679700 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679566    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerName="cleanup"
Nov 03 20:15:33.679700 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679573    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerName="cleanup"
Nov 03 20:15:33.679700 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.679578    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b61be9-c5bf-4d0b-b8ba-d38d721a9457" containerName="cleanup"
Nov 03 20:15:33.684222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.684197    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:33.687568 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.687547    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:15:33.688026 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.687995    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:15:33.691703 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.691678    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"]
Nov 03 20:15:33.716113 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.716080    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:15:33.716113 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.716084    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:15:33.819024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.818988    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dxn4q\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:33.819257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.819200    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dxn4q\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:33.819257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.819237    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqkh\" (UniqueName: \"kubernetes.io/projected/f0f984ea-d900-44a0-b2bd-80ff5903e18e-kube-api-access-glqkh\") pod \"maas-api-revoke-sub-3d4156f1-dxn4q\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:33.920384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.920281    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dxn4q\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:33.920564 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.920389    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dxn4q\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:33.920564 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.920417    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqkh\" (UniqueName: \"kubernetes.io/projected/f0f984ea-d900-44a0-b2bd-80ff5903e18e-kube-api-access-glqkh\") pod \"maas-api-revoke-sub-3d4156f1-dxn4q\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:33.921004 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.920977    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dxn4q\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:33.922887 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.922862    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dxn4q\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:33.930560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:33.930537    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqkh\" (UniqueName: \"kubernetes.io/projected/f0f984ea-d900-44a0-b2bd-80ff5903e18e-kube-api-access-glqkh\") pod \"maas-api-revoke-sub-3d4156f1-dxn4q\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:34.000135 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.000102    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:34.176997 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.176901    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn" podUID="9cf3b7d1-66f0-4a2a-914a-0d459285aa72" containerName="main" containerID="cri-o://b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302" gracePeriod=45
Nov 03 20:15:34.274544 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.274516    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"]
Nov 03 20:15:34.276902 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:15:34.276873    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f984ea_d900_44a0_b2bd_80ff5903e18e.slice/crio-04607b6adb136fe10a34431f0ca631cb0577e8d7ba427ee9e48c7fffc292c3be WatchSource:0}: Error finding container 04607b6adb136fe10a34431f0ca631cb0577e8d7ba427ee9e48c7fffc292c3be: Status 404 returned error can't find the container with id 04607b6adb136fe10a34431f0ca631cb0577e8d7ba427ee9e48c7fffc292c3be
Nov 03 20:15:34.420518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.420494    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:15:34.526052 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.526011    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-kube-api-access-f9n7q\" (UniqueName: \"kubernetes.io/projected/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-kube-api-access-f9n7q\") pod \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") "
Nov 03 20:15:34.526249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.526109    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-home\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-home\") pod \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") "
Nov 03 20:15:34.526249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.526148    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-dshm\") pod \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") "
Nov 03 20:15:34.526249 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.526204    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tls-certs\") pod \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") "
Nov 03 20:15:34.526452 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.526319    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tmp-dir\") pod \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") "
Nov 03 20:15:34.526452 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.526355    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-model-cache\") pod \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\" (UID: \"9cf3b7d1-66f0-4a2a-914a-0d459285aa72\") "
Nov 03 20:15:34.526452 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.526434    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-home" pod "9cf3b7d1-66f0-4a2a-914a-0d459285aa72" (UID: "9cf3b7d1-66f0-4a2a-914a-0d459285aa72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:34.526639 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.526604    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tmp-dir" pod "9cf3b7d1-66f0-4a2a-914a-0d459285aa72" (UID: "9cf3b7d1-66f0-4a2a-914a-0d459285aa72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:34.526820 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.526687    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-model-cache" pod "9cf3b7d1-66f0-4a2a-914a-0d459285aa72" (UID: "9cf3b7d1-66f0-4a2a-914a-0d459285aa72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:34.526820 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.526702    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:34.528735 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.528704    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-dshm" pod "9cf3b7d1-66f0-4a2a-914a-0d459285aa72" (UID: "9cf3b7d1-66f0-4a2a-914a-0d459285aa72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:34.528735 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.528705    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tls-certs" pod "9cf3b7d1-66f0-4a2a-914a-0d459285aa72" (UID: "9cf3b7d1-66f0-4a2a-914a-0d459285aa72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:15:34.528902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.528743    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-kube-api-access-f9n7q" pod "9cf3b7d1-66f0-4a2a-914a-0d459285aa72" (UID: "9cf3b7d1-66f0-4a2a-914a-0d459285aa72"). InnerVolumeSpecName "kube-api-access-f9n7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:34.627791 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.627755    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:34.627791 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.627785    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:34.627791 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.627796    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9n7q\" (UniqueName: \"kubernetes.io/projected/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-kube-api-access-f9n7q\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:34.628095 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.627805    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:34.628095 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.627813    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf3b7d1-66f0-4a2a-914a-0d459285aa72-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:34.860350 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.860260    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4123ea7d-bdbf-4d51-ac79-e9612073a5c9" path="/var/lib/kubelet/pods/4123ea7d-bdbf-4d51-ac79-e9612073a5c9/volumes"
Nov 03 20:15:34.969262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.969224    4531 generic.go:356] "Generic (PLEG): container finished" podID="9cf3b7d1-66f0-4a2a-914a-0d459285aa72" containerID="b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302" exitCode=0
Nov 03 20:15:34.969262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.969268    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"
Nov 03 20:15:34.969539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.969300    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn" event={"ID":"9cf3b7d1-66f0-4a2a-914a-0d459285aa72","Type":"ContainerDied","Data":"b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302"}
Nov 03 20:15:34.969539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.969368    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn" event={"ID":"9cf3b7d1-66f0-4a2a-914a-0d459285aa72","Type":"ContainerDied","Data":"53e849a9f167ed64cbaaebd13ef5663cf6fd38cc745b3fafb0936151847145b3"}
Nov 03 20:15:34.969539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.969392    4531 scope.go:122] "RemoveContainer" containerID="b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302"
Nov 03 20:15:34.971118 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.971084    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q" event={"ID":"f0f984ea-d900-44a0-b2bd-80ff5903e18e","Type":"ContainerStarted","Data":"ca03cdf37bc63b0716bcfa6a84402f1f4e269eef28462e08f86055a525331039"}
Nov 03 20:15:34.971252 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.971126    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q" event={"ID":"f0f984ea-d900-44a0-b2bd-80ff5903e18e","Type":"ContainerStarted","Data":"04607b6adb136fe10a34431f0ca631cb0577e8d7ba427ee9e48c7fffc292c3be"}
Nov 03 20:15:34.979561 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.979539    4531 scope.go:122] "RemoveContainer" containerID="b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302"
Nov 03 20:15:34.979881 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:15:34.979862    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302\": container with ID starting with b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302 not found: ID does not exist" containerID="b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302"
Nov 03 20:15:34.979934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.979890    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302"} err="failed to get container status \"b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302\": rpc error: code = NotFound desc = could not find container \"b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302\": container with ID starting with b04c867c9624b6621222fa24ffa7964f0d69d6e5f09d1ae5b72c929b222eb302 not found: ID does not exist"
Nov 03 20:15:34.985739 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.985711    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"]
Nov 03 20:15:34.988902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:34.988873    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-030599/simulator-single-main-030599-kserve-5c6f6c947d-wxmnn"]
Nov 03 20:15:35.003645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:35.003565    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q" podStartSLOduration=2.003546449 podStartE2EDuration="2.003546449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:35.000161559 +0000 UTC m=+2648.270284874" watchObservedRunningTime="2026-11-03 20:15:35.003546449 +0000 UTC m=+2648.273669743"
Nov 03 20:15:35.978890 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:35.978851    4531 generic.go:356] "Generic (PLEG): container finished" podID="f0f984ea-d900-44a0-b2bd-80ff5903e18e" containerID="ca03cdf37bc63b0716bcfa6a84402f1f4e269eef28462e08f86055a525331039" exitCode=0
Nov 03 20:15:35.979311 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:35.978909    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q" event={"ID":"f0f984ea-d900-44a0-b2bd-80ff5903e18e","Type":"ContainerDied","Data":"ca03cdf37bc63b0716bcfa6a84402f1f4e269eef28462e08f86055a525331039"}
Nov 03 20:15:36.523593 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:36.523557    4531 scope.go:122] "RemoveContainer" containerID="6f980099b0729370ab11c0c3043f765bdac3b76697a8dbeff1a7dd718ad248ca"
Nov 03 20:15:36.532617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:36.532593    4531 scope.go:122] "RemoveContainer" containerID="6e42a53bf621650b6a4ba42051bc7b019f8f0d4f4e56ef9d89dff5f6e4e53304"
Nov 03 20:15:36.857142 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:36.857057    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf3b7d1-66f0-4a2a-914a-0d459285aa72" path="/var/lib/kubelet/pods/9cf3b7d1-66f0-4a2a-914a-0d459285aa72/volumes"
Nov 03 20:15:37.130140 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.130115    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:37.261560 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.261514    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-openshift-ca\") pod \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") "
Nov 03 20:15:37.261757 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.261584    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-xks-ca\") pod \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") "
Nov 03 20:15:37.261822 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.261746    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f984ea-d900-44a0-b2bd-80ff5903e18e-kube-api-access-glqkh\" (UniqueName: \"kubernetes.io/projected/f0f984ea-d900-44a0-b2bd-80ff5903e18e-kube-api-access-glqkh\") pod \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\" (UID: \"f0f984ea-d900-44a0-b2bd-80ff5903e18e\") "
Nov 03 20:15:37.261923 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.261899    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-openshift-ca" pod "f0f984ea-d900-44a0-b2bd-80ff5903e18e" (UID: "f0f984ea-d900-44a0-b2bd-80ff5903e18e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:37.262138 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.262120    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.263701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.263679    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f984ea-d900-44a0-b2bd-80ff5903e18e-kube-api-access-glqkh" pod "f0f984ea-d900-44a0-b2bd-80ff5903e18e" (UID: "f0f984ea-d900-44a0-b2bd-80ff5903e18e"). InnerVolumeSpecName "kube-api-access-glqkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:37.263793 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.263774    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-xks-ca" pod "f0f984ea-d900-44a0-b2bd-80ff5903e18e" (UID: "f0f984ea-d900-44a0-b2bd-80ff5903e18e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:15:37.363254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.363137    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glqkh\" (UniqueName: \"kubernetes.io/projected/f0f984ea-d900-44a0-b2bd-80ff5903e18e-kube-api-access-glqkh\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.363254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.363195    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f984ea-d900-44a0-b2bd-80ff5903e18e-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.525487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.525457    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body=
Nov 03 20:15:37.525609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.525513    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused"
Nov 03 20:15:37.728443 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.728415    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:15:37.869110 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869072    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-credential-socket\") pod \"5d6cfe10-1e23-484c-a481-91f2daba30a4\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") "
Nov 03 20:15:37.869354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869161    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-podinfo\") pod \"5d6cfe10-1e23-484c-a481-91f2daba30a4\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") "
Nov 03 20:15:37.869354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869219    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-token\" (UniqueName: \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-token\") pod \"5d6cfe10-1e23-484c-a481-91f2daba30a4\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") "
Nov 03 20:15:37.869354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869247    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-data\") pod \"5d6cfe10-1e23-484c-a481-91f2daba30a4\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") "
Nov 03 20:15:37.869354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869275    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-certs\") pod \"5d6cfe10-1e23-484c-a481-91f2daba30a4\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") "
Nov 03 20:15:37.869354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869320    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-envoy\") pod \"5d6cfe10-1e23-484c-a481-91f2daba30a4\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") "
Nov 03 20:15:37.869354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869345    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-socket\") pod \"5d6cfe10-1e23-484c-a481-91f2daba30a4\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") "
Nov 03 20:15:37.869709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869404    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-kube-api-access-dsljh\" (UniqueName: \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-kube-api-access-dsljh\") pod \"5d6cfe10-1e23-484c-a481-91f2daba30a4\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") "
Nov 03 20:15:37.869709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869428    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6cfe10-1e23-484c-a481-91f2daba30a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6cfe10-1e23-484c-a481-91f2daba30a4-istiod-ca-cert\") pod \"5d6cfe10-1e23-484c-a481-91f2daba30a4\" (UID: \"5d6cfe10-1e23-484c-a481-91f2daba30a4\") "
Nov 03 20:15:37.869709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869453    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-credential-socket" pod "5d6cfe10-1e23-484c-a481-91f2daba30a4" (UID: "5d6cfe10-1e23-484c-a481-91f2daba30a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:37.869709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869680    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-data" pod "5d6cfe10-1e23-484c-a481-91f2daba30a4" (UID: "5d6cfe10-1e23-484c-a481-91f2daba30a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:37.869709 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869698    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-socket" pod "5d6cfe10-1e23-484c-a481-91f2daba30a4" (UID: "5d6cfe10-1e23-484c-a481-91f2daba30a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:37.870019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869827    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.870019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869847    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.870019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.869968    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.870217 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.870072    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-certs" pod "5d6cfe10-1e23-484c-a481-91f2daba30a4" (UID: "5d6cfe10-1e23-484c-a481-91f2daba30a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:37.870217 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.870101    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6cfe10-1e23-484c-a481-91f2daba30a4-istiod-ca-cert" pod "5d6cfe10-1e23-484c-a481-91f2daba30a4" (UID: "5d6cfe10-1e23-484c-a481-91f2daba30a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:37.871719 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.871691    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-envoy" pod "5d6cfe10-1e23-484c-a481-91f2daba30a4" (UID: "5d6cfe10-1e23-484c-a481-91f2daba30a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:37.871841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.871770    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-kube-api-access-dsljh" pod "5d6cfe10-1e23-484c-a481-91f2daba30a4" (UID: "5d6cfe10-1e23-484c-a481-91f2daba30a4"). InnerVolumeSpecName "kube-api-access-dsljh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:37.871970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.871949    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-podinfo" pod "5d6cfe10-1e23-484c-a481-91f2daba30a4" (UID: "5d6cfe10-1e23-484c-a481-91f2daba30a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:15:37.872013 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.871963    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-token" pod "5d6cfe10-1e23-484c-a481-91f2daba30a4" (UID: "5d6cfe10-1e23-484c-a481-91f2daba30a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:37.971090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.970984    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.971090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.971023    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.971090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.971038    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.971090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.971052    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6cfe10-1e23-484c-a481-91f2daba30a4-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.971090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.971068    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsljh\" (UniqueName: \"kubernetes.io/projected/5d6cfe10-1e23-484c-a481-91f2daba30a4-kube-api-access-dsljh\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.971445 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.971101    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6cfe10-1e23-484c-a481-91f2daba30a4-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:37.991786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.991747    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"
Nov 03 20:15:37.991786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.991761    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q" event={"ID":"f0f984ea-d900-44a0-b2bd-80ff5903e18e","Type":"ContainerDied","Data":"04607b6adb136fe10a34431f0ca631cb0577e8d7ba427ee9e48c7fffc292c3be"}
Nov 03 20:15:37.992001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.991814    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04607b6adb136fe10a34431f0ca631cb0577e8d7ba427ee9e48c7fffc292c3be"
Nov 03 20:15:37.993657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.993634    4531 generic.go:356] "Generic (PLEG): container finished" podID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerID="93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e" exitCode=0
Nov 03 20:15:37.993798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.993693    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" event={"ID":"5d6cfe10-1e23-484c-a481-91f2daba30a4","Type":"ContainerDied","Data":"93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e"}
Nov 03 20:15:37.993798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.993695    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"
Nov 03 20:15:37.993798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.993727    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj" event={"ID":"5d6cfe10-1e23-484c-a481-91f2daba30a4","Type":"ContainerDied","Data":"d0c139c65eafbd56835c6ba664bae5912498ad74aceb9097900a9322bd348097"}
Nov 03 20:15:37.993798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:37.993749    4531 scope.go:122] "RemoveContainer" containerID="93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e"
Nov 03 20:15:38.004719 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:38.004693    4531 scope.go:122] "RemoveContainer" containerID="145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc"
Nov 03 20:15:38.013660 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:38.013642    4531 scope.go:122] "RemoveContainer" containerID="93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e"
Nov 03 20:15:38.013950 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:15:38.013930    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e\": container with ID starting with 93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e not found: ID does not exist" containerID="93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e"
Nov 03 20:15:38.013998 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:38.013961    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e"} err="failed to get container status \"93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e\": rpc error: code = NotFound desc = could not find container \"93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e\": container with ID starting with 93d7cfabbeb49704d576141b3b7ca8b9f4aa87f7566f26ad3110c0e6bf7af75e not found: ID does not exist"
Nov 03 20:15:38.013998 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:38.013979    4531 scope.go:122] "RemoveContainer" containerID="145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc"
Nov 03 20:15:38.014244 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:15:38.014224    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc\": container with ID starting with 145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc not found: ID does not exist" containerID="145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc"
Nov 03 20:15:38.014292 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:38.014252    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc"} err="failed to get container status \"145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc\": rpc error: code = NotFound desc = could not find container \"145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc\": container with ID starting with 145844854ce5dd58c014955cd88e7aef2e26ee3955a871119af76d189ec77adc not found: ID does not exist"
Nov 03 20:15:38.018932 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:38.018907    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"]
Nov 03 20:15:38.021915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:38.021896    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-030599-openshift-default-f6d7855d5-8zbjj"]
Nov 03 20:15:38.857436 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:38.857398    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" path="/var/lib/kubelet/pods/5d6cfe10-1e23-484c-a481-91f2daba30a4/volumes"
Nov 03 20:15:39.022281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:39.022242    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"]
Nov 03 20:15:39.023962 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:39.023931    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rl2nr"]
Nov 03 20:15:40.446193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:40.446088    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz" podUID="dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:15:40.446193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:40.446117    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8" podUID="48c3a53b-edaf-447c-948f-d498a3dd966b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:15:40.857066 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:40.856990    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee744b4-c817-42d0-8b8d-74cb9da5c4de" path="/var/lib/kubelet/pods/3ee744b4-c817-42d0-8b8d-74cb9da5c4de/volumes"
Nov 03 20:15:49.846888 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.846858    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:15:49.849911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.849886    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:15:49.880915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.880882    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c3a53b-edaf-447c-948f-d498a3dd966b-kube-api-access-kpjln\" (UniqueName: \"kubernetes.io/projected/48c3a53b-edaf-447c-948f-d498a3dd966b-kube-api-access-kpjln\") pod \"48c3a53b-edaf-447c-948f-d498a3dd966b\" (UID: \"48c3a53b-edaf-447c-948f-d498a3dd966b\") "
Nov 03 20:15:49.881260 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.881235    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-plugins-config-volume\") pod \"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d\" (UID: \"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d\") "
Nov 03 20:15:49.881449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.881425    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-kube-api-access-2ktf7\" (UniqueName: \"kubernetes.io/projected/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-kube-api-access-2ktf7\") pod \"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d\" (UID: \"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d\") "
Nov 03 20:15:49.881582 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.881465    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c3a53b-edaf-447c-948f-d498a3dd966b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c3a53b-edaf-447c-948f-d498a3dd966b-plugins-config-volume\") pod \"48c3a53b-edaf-447c-948f-d498a3dd966b\" (UID: \"48c3a53b-edaf-447c-948f-d498a3dd966b\") "
Nov 03 20:15:49.881658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.881630    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-plugins-config-volume" pod "dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d" (UID: "dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:49.882019 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.881993    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c3a53b-edaf-447c-948f-d498a3dd966b-plugins-config-volume" pod "48c3a53b-edaf-447c-948f-d498a3dd966b" (UID: "48c3a53b-edaf-447c-948f-d498a3dd966b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:49.882146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.882128    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:49.882256 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.882149    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c3a53b-edaf-447c-948f-d498a3dd966b-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:49.883500 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.883477    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c3a53b-edaf-447c-948f-d498a3dd966b-kube-api-access-kpjln" pod "48c3a53b-edaf-447c-948f-d498a3dd966b" (UID: "48c3a53b-edaf-447c-948f-d498a3dd966b"). InnerVolumeSpecName "kube-api-access-kpjln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:49.884672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.884649    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-kube-api-access-2ktf7" pod "dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d" (UID: "dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d"). InnerVolumeSpecName "kube-api-access-2ktf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:49.982983 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.982945    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpjln\" (UniqueName: \"kubernetes.io/projected/48c3a53b-edaf-447c-948f-d498a3dd966b-kube-api-access-kpjln\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:49.982983 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:49.982977    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ktf7\" (UniqueName: \"kubernetes.io/projected/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d-kube-api-access-2ktf7\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:50.019930 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.019896    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"]
Nov 03 20:15:50.022186 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.022148    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwr7w"]
Nov 03 20:15:50.060364 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.060321    4531 generic.go:356] "Generic (PLEG): container finished" podID="dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d" containerID="86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e" exitCode=0
Nov 03 20:15:50.060573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.060394    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"
Nov 03 20:15:50.060573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.060399    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz" event={"ID":"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d","Type":"ContainerDied","Data":"86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e"}
Nov 03 20:15:50.060573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.060499    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz" event={"ID":"dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d","Type":"ContainerDied","Data":"5434a49da7fa8a901643ca0c3f85be4af774c5311de6ef52b68b004cc2ac82c4"}
Nov 03 20:15:50.060573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.060532    4531 scope.go:122] "RemoveContainer" containerID="86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e"
Nov 03 20:15:50.061825 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.061798    4531 generic.go:356] "Generic (PLEG): container finished" podID="48c3a53b-edaf-447c-948f-d498a3dd966b" containerID="ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb" exitCode=0
Nov 03 20:15:50.061825 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.061828    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8" event={"ID":"48c3a53b-edaf-447c-948f-d498a3dd966b","Type":"ContainerDied","Data":"ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb"}
Nov 03 20:15:50.062028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.061863    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"
Nov 03 20:15:50.062028 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.061872    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8" event={"ID":"48c3a53b-edaf-447c-948f-d498a3dd966b","Type":"ContainerDied","Data":"c24d02f8484a11f565b44187ee528453e57ae1ecd0ea9d6b3004efb0fbf0caf2"}
Nov 03 20:15:50.070109 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.070088    4531 scope.go:122] "RemoveContainer" containerID="86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e"
Nov 03 20:15:50.070440 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:15:50.070422    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e\": container with ID starting with 86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e not found: ID does not exist" containerID="86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e"
Nov 03 20:15:50.070502 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.070450    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e"} err="failed to get container status \"86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e\": rpc error: code = NotFound desc = could not find container \"86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e\": container with ID starting with 86f5abd055f84bdbd53dc0f2cb96f284f53e9265e8e579ee4104773e7a5fb85e not found: ID does not exist"
Nov 03 20:15:50.070502 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.070467    4531 scope.go:122] "RemoveContainer" containerID="ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb"
Nov 03 20:15:50.079510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.079491    4531 scope.go:122] "RemoveContainer" containerID="ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb"
Nov 03 20:15:50.079792 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:15:50.079770    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb\": container with ID starting with ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb not found: ID does not exist" containerID="ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb"
Nov 03 20:15:50.079853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.079799    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb"} err="failed to get container status \"ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb\": rpc error: code = NotFound desc = could not find container \"ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb\": container with ID starting with ededd51cb7e087af3681a3e78b8dfcfaf52d1eec5d692f55a5eea969df5c8acb not found: ID does not exist"
Nov 03 20:15:50.083947 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.083921    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"]
Nov 03 20:15:50.086196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.086161    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-030599-55c7b56b7htbnz"]
Nov 03 20:15:50.094014 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.093991    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"]
Nov 03 20:15:50.096119 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.096098    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-030599-5545bxwwg8"]
Nov 03 20:15:50.574048 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:15:50.574003    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:15:50.857466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.857375    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c3a53b-edaf-447c-948f-d498a3dd966b" path="/var/lib/kubelet/pods/48c3a53b-edaf-447c-948f-d498a3dd966b/volumes"
Nov 03 20:15:50.857970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.857947    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d" path="/var/lib/kubelet/pods/dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d/volumes"
Nov 03 20:15:50.858429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:50.858408    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f761666e-87a9-4ff6-9309-490667dcd1c8" path="/var/lib/kubelet/pods/f761666e-87a9-4ff6-9309-490667dcd1c8/volumes"
Nov 03 20:15:55.633476 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:55.633435    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:56.414282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:15:56.414243    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:01.953417 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:01.953377    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:17.018885 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:17.018824    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"]
Nov 03 20:16:17.020959 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:17.020931    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7tdjq"]
Nov 03 20:16:17.545516 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:17.545477    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:18.856658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:18.856623    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d241449b-a313-43ee-bb54-bea78b294161" path="/var/lib/kubelet/pods/d241449b-a313-43ee-bb54-bea78b294161/volumes"
Nov 03 20:16:20.572106 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:16:20.572069    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:16:26.737924 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:26.737886    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:27.020160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:27.020059    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"]
Nov 03 20:16:27.022022 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:27.021995    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hbg9h"]
Nov 03 20:16:28.857031 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:28.856996    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c54234-fb21-4d9c-b1eb-218db58da13b" path="/var/lib/kubelet/pods/70c54234-fb21-4d9c-b1eb-218db58da13b/volumes"
Nov 03 20:16:32.594133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:32.594100    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:16:32.625671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:32.625644    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:16:36.570868 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:36.570832    4531 scope.go:122] "RemoveContainer" containerID="46cbb80ec590131a87cb0df90ab5ffbb24796566e9ab02779a79f96605fbfff3"
Nov 03 20:16:36.579870 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:36.579847    4531 scope.go:122] "RemoveContainer" containerID="3a97ad7c7dcf7a681cb04f07d77592729295cac894f69e1065047fd4c424f795"
Nov 03 20:16:47.020462 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:47.020420    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"]
Nov 03 20:16:47.022509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:47.022481    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9952m"]
Nov 03 20:16:47.378271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:47.378165    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:48.857649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:48.857613    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afa88be-a289-4c8f-b6cc-a370296e1b60" path="/var/lib/kubelet/pods/9afa88be-a289-4c8f-b6cc-a370296e1b60/volumes"
Nov 03 20:16:50.572977 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:16:50.572937    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:16:56.484404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:16:56.484365    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:05.019131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:05.019092    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"]
Nov 03 20:17:05.023110 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:05.023085    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hjbpw"]
Nov 03 20:17:06.856732 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:06.856698    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d39ddc-600d-4d84-9ca8-42144a6e6b0d" path="/var/lib/kubelet/pods/31d39ddc-600d-4d84-9ca8-42144a6e6b0d/volumes"
Nov 03 20:17:11.746920 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.746836    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"]
Nov 03 20:17:11.747315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747250    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy"
Nov 03 20:17:11.747315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747261    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy"
Nov 03 20:17:11.747315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747275    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d" containerName="payload-processing"
Nov 03 20:17:11.747315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747280    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d" containerName="payload-processing"
Nov 03 20:17:11.747315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747308    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9cf3b7d1-66f0-4a2a-914a-0d459285aa72" containerName="main"
Nov 03 20:17:11.747315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747313    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf3b7d1-66f0-4a2a-914a-0d459285aa72" containerName="main"
Nov 03 20:17:11.747531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747322    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f0f984ea-d900-44a0-b2bd-80ff5903e18e" containerName="revoke-keys"
Nov 03 20:17:11.747531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747327    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f984ea-d900-44a0-b2bd-80ff5903e18e" containerName="revoke-keys"
Nov 03 20:17:11.747531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747334    4531 cpu_manager.go:414] "containerMap: removing container" podUID="48c3a53b-edaf-447c-948f-d498a3dd966b" containerName="payload-pre-processing"
Nov 03 20:17:11.747531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747339    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c3a53b-edaf-447c-948f-d498a3dd966b" containerName="payload-pre-processing"
Nov 03 20:17:11.747531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747392    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf3b7d1-66f0-4a2a-914a-0d459285aa72" containerName="main"
Nov 03 20:17:11.747531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747401    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f984ea-d900-44a0-b2bd-80ff5903e18e" containerName="revoke-keys"
Nov 03 20:17:11.747531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747407    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy"
Nov 03 20:17:11.747531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747412    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c3a53b-edaf-447c-948f-d498a3dd966b" containerName="payload-pre-processing"
Nov 03 20:17:11.747531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747419    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy"
Nov 03 20:17:11.747531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.747424    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4bc543-ceea-49fc-a3e7-68ad7dcefb6d" containerName="payload-processing"
Nov 03 20:17:11.750823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.750802    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:11.756129 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.756101    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:17:11.756359 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.756336    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:17:11.756485 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.756469    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:17:11.756557 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.756517    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:17:11.774752 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.774718    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"]
Nov 03 20:17:11.848589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.848544    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2cddl\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:11.848786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.848605    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2cddl\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:11.848786 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.848663    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9xt\" (UniqueName: \"kubernetes.io/projected/59dab081-a07a-4014-9a4c-8a204510cb1a-kube-api-access-bx9xt\") pod \"maas-api-revoke-sub-d576e68b-2cddl\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:11.950079 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.950016    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2cddl\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:11.950079 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.950075    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9xt\" (UniqueName: \"kubernetes.io/projected/59dab081-a07a-4014-9a4c-8a204510cb1a-kube-api-access-bx9xt\") pod \"maas-api-revoke-sub-d576e68b-2cddl\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:11.950413 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.950276    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2cddl\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:11.951026 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.950998    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2cddl\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:11.952508 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.952486    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2cddl\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:11.957103 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:11.957079    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9xt\" (UniqueName: \"kubernetes.io/projected/59dab081-a07a-4014-9a4c-8a204510cb1a-kube-api-access-bx9xt\") pod \"maas-api-revoke-sub-d576e68b-2cddl\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:12.074435 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:12.074340    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:12.345446 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:12.345417    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"]
Nov 03 20:17:12.347670 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:17:12.347639    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dab081_a07a_4014_9a4c_8a204510cb1a.slice/crio-718e3fdcd9e8cf6c5ae0077d905ddaeda2104c7cd975db3cf94aa6e129c55541 WatchSource:0}: Error finding container 718e3fdcd9e8cf6c5ae0077d905ddaeda2104c7cd975db3cf94aa6e129c55541: Status 404 returned error can't find the container with id 718e3fdcd9e8cf6c5ae0077d905ddaeda2104c7cd975db3cf94aa6e129c55541
Nov 03 20:17:12.510098 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:12.510059    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl" event={"ID":"59dab081-a07a-4014-9a4c-8a204510cb1a","Type":"ContainerStarted","Data":"718e3fdcd9e8cf6c5ae0077d905ddaeda2104c7cd975db3cf94aa6e129c55541"}
Nov 03 20:17:13.516760 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:13.516678    4531 generic.go:356] "Generic (PLEG): container finished" podID="59dab081-a07a-4014-9a4c-8a204510cb1a" containerID="1e35991efdf03c7be93f72454fb344d9f849eacd435e34d4706867ed6077f9e1" exitCode=0
Nov 03 20:17:13.517160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:13.516759    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl" event={"ID":"59dab081-a07a-4014-9a4c-8a204510cb1a","Type":"ContainerDied","Data":"1e35991efdf03c7be93f72454fb344d9f849eacd435e34d4706867ed6077f9e1"}
Nov 03 20:17:14.653067 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:14.653040    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:14.776055 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:14.775960    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-openshift-ca\") pod \"59dab081-a07a-4014-9a4c-8a204510cb1a\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") "
Nov 03 20:17:14.776271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:14.776065    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59dab081-a07a-4014-9a4c-8a204510cb1a-kube-api-access-bx9xt\" (UniqueName: \"kubernetes.io/projected/59dab081-a07a-4014-9a4c-8a204510cb1a-kube-api-access-bx9xt\") pod \"59dab081-a07a-4014-9a4c-8a204510cb1a\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") "
Nov 03 20:17:14.776271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:14.776110    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-xks-ca\") pod \"59dab081-a07a-4014-9a4c-8a204510cb1a\" (UID: \"59dab081-a07a-4014-9a4c-8a204510cb1a\") "
Nov 03 20:17:14.776456 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:14.776431    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-openshift-ca" pod "59dab081-a07a-4014-9a4c-8a204510cb1a" (UID: "59dab081-a07a-4014-9a4c-8a204510cb1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:14.778193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:14.778157    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-xks-ca" pod "59dab081-a07a-4014-9a4c-8a204510cb1a" (UID: "59dab081-a07a-4014-9a4c-8a204510cb1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:17:14.778273 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:14.778250    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dab081-a07a-4014-9a4c-8a204510cb1a-kube-api-access-bx9xt" pod "59dab081-a07a-4014-9a4c-8a204510cb1a" (UID: "59dab081-a07a-4014-9a4c-8a204510cb1a"). InnerVolumeSpecName "kube-api-access-bx9xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:14.877833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:14.877802    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx9xt\" (UniqueName: \"kubernetes.io/projected/59dab081-a07a-4014-9a4c-8a204510cb1a-kube-api-access-bx9xt\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.877833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:14.877831    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.878037 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:14.877841    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59dab081-a07a-4014-9a4c-8a204510cb1a-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:15.528158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:15.528122    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"
Nov 03 20:17:15.528158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:15.528136    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl" event={"ID":"59dab081-a07a-4014-9a4c-8a204510cb1a","Type":"ContainerDied","Data":"718e3fdcd9e8cf6c5ae0077d905ddaeda2104c7cd975db3cf94aa6e129c55541"}
Nov 03 20:17:15.528445 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:15.528192    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718e3fdcd9e8cf6c5ae0077d905ddaeda2104c7cd975db3cf94aa6e129c55541"
Nov 03 20:17:17.637878 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:17.637847    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:20.573198 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:17:20.573138    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:17:24.019768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:24.019730    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"]
Nov 03 20:17:24.021945 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:24.021916    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-9slrb"]
Nov 03 20:17:24.860550 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:24.860502    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f074e13-c713-4e39-8780-b452c3c8b176" path="/var/lib/kubelet/pods/8f074e13-c713-4e39-8780-b452c3c8b176/volumes"
Nov 03 20:17:30.397631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.397594    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"]
Nov 03 20:17:30.398153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.397996    4531 cpu_manager.go:414] "containerMap: removing container" podUID="59dab081-a07a-4014-9a4c-8a204510cb1a" containerName="revoke-keys"
Nov 03 20:17:30.398153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.398007    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59dab081-a07a-4014-9a4c-8a204510cb1a" containerName="revoke-keys"
Nov 03 20:17:30.398153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.398034    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy"
Nov 03 20:17:30.398153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.398039    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6cfe10-1e23-484c-a481-91f2daba30a4" containerName="istio-proxy"
Nov 03 20:17:30.398153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.398114    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="59dab081-a07a-4014-9a4c-8a204510cb1a" containerName="revoke-keys"
Nov 03 20:17:30.402920 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.402882    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:30.406343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.406315    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:17:30.406618 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.406599    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:17:30.406868 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.406841    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:17:30.407332 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.407263    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:17:30.409643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.409620    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"]
Nov 03 20:17:30.464928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.464883    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghtf5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:30.465124 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.464948    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tkb6\" (UniqueName: \"kubernetes.io/projected/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-kube-api-access-6tkb6\") pod \"maas-api-revoke-sub-d2bc93bc-ghtf5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:30.465124 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.465050    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghtf5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:30.566306 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.566265    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghtf5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:30.566486 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.566332    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tkb6\" (UniqueName: \"kubernetes.io/projected/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-kube-api-access-6tkb6\") pod \"maas-api-revoke-sub-d2bc93bc-ghtf5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:30.566486 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.566373    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghtf5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:30.567060 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.567037    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghtf5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:30.568795 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.568776    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghtf5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:30.573282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.573253    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tkb6\" (UniqueName: \"kubernetes.io/projected/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-kube-api-access-6tkb6\") pod \"maas-api-revoke-sub-d2bc93bc-ghtf5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:30.721657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.721556    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:30.995327 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:30.995300    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"]
Nov 03 20:17:30.998307 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:17:30.998263    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb447e4d8_0aed_4ae9_938c_ca9d8c6a8ac5.slice/crio-a97171082383fbeb9ff1a967af60ef5a9b26c1257e9aec8bf624a8ee883b4dc0 WatchSource:0}: Error finding container a97171082383fbeb9ff1a967af60ef5a9b26c1257e9aec8bf624a8ee883b4dc0: Status 404 returned error can't find the container with id a97171082383fbeb9ff1a967af60ef5a9b26c1257e9aec8bf624a8ee883b4dc0
Nov 03 20:17:31.650734 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:31.650698    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5" event={"ID":"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5","Type":"ContainerStarted","Data":"16d4a3bd61df05607c842c51801b4708b2d08fbbb7b1fc278c6a2ee768e38080"}
Nov 03 20:17:31.651207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:31.650755    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5" event={"ID":"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5","Type":"ContainerStarted","Data":"a97171082383fbeb9ff1a967af60ef5a9b26c1257e9aec8bf624a8ee883b4dc0"}
Nov 03 20:17:31.668650 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:31.668595    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5" podStartSLOduration=1.6685806090000002 podStartE2EDuration="1.668580609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:31.665602925 +0000 UTC m=+2764.935726219" watchObservedRunningTime="2026-11-03 20:17:31.668580609 +0000 UTC m=+2764.938703972"
Nov 03 20:17:32.656814 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:32.656779    4531 generic.go:356] "Generic (PLEG): container finished" podID="b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5" containerID="16d4a3bd61df05607c842c51801b4708b2d08fbbb7b1fc278c6a2ee768e38080" exitCode=0
Nov 03 20:17:32.657265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:32.656844    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5" event={"ID":"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5","Type":"ContainerDied","Data":"16d4a3bd61df05607c842c51801b4708b2d08fbbb7b1fc278c6a2ee768e38080"}
Nov 03 20:17:33.801157 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:33.801127    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:33.904287 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:33.904249    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-xks-ca\") pod \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") "
Nov 03 20:17:33.904446 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:33.904299    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-openshift-ca\") pod \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") "
Nov 03 20:17:33.904446 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:33.904342    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-kube-api-access-6tkb6\" (UniqueName: \"kubernetes.io/projected/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-kube-api-access-6tkb6\") pod \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\" (UID: \"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5\") "
Nov 03 20:17:33.904725 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:33.904699    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-openshift-ca" pod "b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5" (UID: "b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:33.906503 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:33.906477    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-xks-ca" pod "b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5" (UID: "b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:17:33.906601 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:33.906551    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-kube-api-access-6tkb6" pod "b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5" (UID: "b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5"). InnerVolumeSpecName "kube-api-access-6tkb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:34.005383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:34.005350    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:34.005383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:34.005378    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tkb6\" (UniqueName: \"kubernetes.io/projected/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-kube-api-access-6tkb6\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:34.005383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:34.005388    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:34.668346 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:34.668314    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"
Nov 03 20:17:34.668346 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:34.668327    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5" event={"ID":"b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5","Type":"ContainerDied","Data":"a97171082383fbeb9ff1a967af60ef5a9b26c1257e9aec8bf624a8ee883b4dc0"}
Nov 03 20:17:34.668578 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:34.668375    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a97171082383fbeb9ff1a967af60ef5a9b26c1257e9aec8bf624a8ee883b4dc0"
Nov 03 20:17:36.622165 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:36.622129    4531 scope.go:122] "RemoveContainer" containerID="ca25caed94a7fd1d22db213e42ab5c88acfd1a1e0c092471672f06d51ec497d4"
Nov 03 20:17:36.631330 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:36.631298    4531 scope.go:122] "RemoveContainer" containerID="5e71a7fd70fac3585736df5196abc8edbe7fc35da9a58ff08b4f81a46bde0e2a"
Nov 03 20:17:36.639435 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:36.639404    4531 scope.go:122] "RemoveContainer" containerID="013bd4c59f99dd49e5dd5c304e0295e7acf6ef7354fc1b2b6911d07682b2e0ac"
Nov 03 20:17:41.020618 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:41.020579    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"]
Nov 03 20:17:41.022846 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:41.022818    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-k9sxr"]
Nov 03 20:17:41.982415 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:41.982381    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:42.364469 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.364362    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"]
Nov 03 20:17:42.365039 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.364779    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5" containerName="revoke-keys"
Nov 03 20:17:42.365039 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.364791    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5" containerName="revoke-keys"
Nov 03 20:17:42.365039 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.364869    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5" containerName="revoke-keys"
Nov 03 20:17:42.369881 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.369857    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.382540 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.382512    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"]
Nov 03 20:17:42.383539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.383514    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-794e97-openshift-default-dockercfg-p69lk\""
Nov 03 20:17:42.475383 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.475348    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:42.485502 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.485462    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-podinfo\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.485701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.485512    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-token\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.485701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.485549    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istiod-ca-cert\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.485701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.485564    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-envoy\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.485701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.485605    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-credential-socket\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.485701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.485654    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-data\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.485921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.485715    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-certs\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.485921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.485737    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sj6h\" (UniqueName: \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-kube-api-access-2sj6h\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.485921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.485757    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-socket\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.586581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.586539    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-podinfo\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.586581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.586586    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-token\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.586823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.586625    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istiod-ca-cert\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.586823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.586649    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-envoy\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.586823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.586694    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-credential-socket\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.586823 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.586757    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-data\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.587043 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.586826    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-certs\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.587043 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.586861    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sj6h\" (UniqueName: \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-kube-api-access-2sj6h\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.587043 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.586884    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-socket\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.587310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.587234    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-credential-socket\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.587355 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.587340    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-data\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.587519 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.587499    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istiod-ca-cert\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.587619 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.587588    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-socket\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.587691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.587668    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-certs\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.589131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.589101    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-envoy\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.589312 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.589294    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-podinfo\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.593592 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.593561    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-token\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.593688 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.593667    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sj6h\" (UniqueName: \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-kube-api-access-2sj6h\") pod \"e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") " pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.651157 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.651124    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:42.686146 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.686106    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:42.857571 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.857539    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de72b837-9c6a-4efc-b11f-ccf378eb7f8f" path="/var/lib/kubelet/pods/de72b837-9c6a-4efc-b11f-ccf378eb7f8f/volumes"
Nov 03 20:17:42.963090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.963062    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"]
Nov 03 20:17:42.965394 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:17:42.965364    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2948b4d_ffdd_49ca_84b5_1017eaef163a.slice/crio-49364422d82a9b680f1f22d4cb6a0a3269f7e95d9b1081669d79a12006b8a8df WatchSource:0}: Error finding container 49364422d82a9b680f1f22d4cb6a0a3269f7e95d9b1081669d79a12006b8a8df: Status 404 returned error can't find the container with id 49364422d82a9b680f1f22d4cb6a0a3269f7e95d9b1081669d79a12006b8a8df
Nov 03 20:17:42.971952 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.971916    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:42.972079 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.971988    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:42.972079 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:42.972017    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:43.717896 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:43.717857    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" event={"ID":"b2948b4d-ffdd-49ca-84b5-1017eaef163a","Type":"ContainerStarted","Data":"c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864"}
Nov 03 20:17:43.718328 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:43.717920    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" event={"ID":"b2948b4d-ffdd-49ca-84b5-1017eaef163a","Type":"ContainerStarted","Data":"49364422d82a9b680f1f22d4cb6a0a3269f7e95d9b1081669d79a12006b8a8df"}
Nov 03 20:17:43.737105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:43.737048    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" podStartSLOduration=1.7370308269999999 podStartE2EDuration="1.737030827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:43.734667756 +0000 UTC m=+2777.004791050" watchObservedRunningTime="2026-11-03 20:17:43.737030827 +0000 UTC m=+2777.007154123"
Nov 03 20:17:44.686556 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:44.686516    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:44.691888 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:44.691860    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:44.723885 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:44.723857    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:44.724798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:44.724776    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:17:45.971166 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:45.971126    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:46.122227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:46.122162    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:47.135549 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.135515    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"]
Nov 03 20:17:47.139541 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.139514    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.144938 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.144908    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-794e97-serving-cert\""
Nov 03 20:17:47.150684 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.149990    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:17:47.150684 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.150162    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rj6j4\""
Nov 03 20:17:47.150684 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.150648    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:17:47.157381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.157359    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:17:47.158882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.158861    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"]
Nov 03 20:17:47.162838 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.162817    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"]
Nov 03 20:17:47.167553 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.167460    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:47.167711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.167634    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:17:47.172445 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.172417    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"]
Nov 03 20:17:47.174022 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.173985    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a3ad473a-dockercfg-7pdfh\""
Nov 03 20:17:47.229872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.229831    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.230094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.229908    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.230246 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.230131    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qhp8\" (UniqueName: \"kubernetes.io/projected/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-kube-api-access-9qhp8\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.230246 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.230188    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-metrics-certs\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.232697 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.232670    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"]
Nov 03 20:17:47.236318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.236299    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:17:47.245371 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.245349    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-794e97\""
Nov 03 20:17:47.246187 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.246148    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"]
Nov 03 20:17:47.311078 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.311043    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:47.331346 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.331311    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkt25\" (UniqueName: \"kubernetes.io/projected/c545c925-e23b-4155-8af0-edbdd2af211f-kube-api-access-nkt25\") pod \"payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb\" (UID: \"c545c925-e23b-4155-8af0-edbdd2af211f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:17:47.331539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.331382    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.331539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.331438    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd\" (UID: \"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:17:47.331539 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.331481    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.331734 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.331593    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c545c925-e23b-4155-8af0-edbdd2af211f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb\" (UID: \"c545c925-e23b-4155-8af0-edbdd2af211f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:17:47.331734 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.331718    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qhp8\" (UniqueName: \"kubernetes.io/projected/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-kube-api-access-9qhp8\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.331839 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.331760    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-metrics-certs\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.331911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.331895    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncjqx\" (UniqueName: \"kubernetes.io/projected/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-kube-api-access-ncjqx\") pod \"payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd\" (UID: \"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:17:47.332394 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.332369    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.334142 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.334116    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.334323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.334299    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-metrics-certs\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.338528 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.338507    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qhp8\" (UniqueName: \"kubernetes.io/projected/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-kube-api-access-9qhp8\") pod \"maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.433604 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.433506    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncjqx\" (UniqueName: \"kubernetes.io/projected/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-kube-api-access-ncjqx\") pod \"payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd\" (UID: \"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:17:47.433809 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.433609    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkt25\" (UniqueName: \"kubernetes.io/projected/c545c925-e23b-4155-8af0-edbdd2af211f-kube-api-access-nkt25\") pod \"payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb\" (UID: \"c545c925-e23b-4155-8af0-edbdd2af211f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:17:47.433809 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.433665    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd\" (UID: \"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:17:47.433809 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.433725    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c545c925-e23b-4155-8af0-edbdd2af211f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb\" (UID: \"c545c925-e23b-4155-8af0-edbdd2af211f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:17:47.434503 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.434484    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c545c925-e23b-4155-8af0-edbdd2af211f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb\" (UID: \"c545c925-e23b-4155-8af0-edbdd2af211f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:17:47.434575 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.434501    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd\" (UID: \"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:17:47.437509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.437484    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:47.440806 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.440784    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkt25\" (UniqueName: \"kubernetes.io/projected/c545c925-e23b-4155-8af0-edbdd2af211f-kube-api-access-nkt25\") pod \"payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb\" (UID: \"c545c925-e23b-4155-8af0-edbdd2af211f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:17:47.440943 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.440811    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncjqx\" (UniqueName: \"kubernetes.io/projected/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-kube-api-access-ncjqx\") pod \"payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd\" (UID: \"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:17:47.460655 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.460631    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:47.489613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.489581    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:17:47.554476 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.551892    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:17:47.740370 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.740340    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"]
Nov 03 20:17:47.741362 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:17:47.741334    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6eeea2_b2b3_4a7a_b9a2_50f003198a50.slice/crio-597f6bf151d4a7a213d7a583b2240654bc91fdbf87042735e497635724db8476 WatchSource:0}: Error finding container 597f6bf151d4a7a213d7a583b2240654bc91fdbf87042735e497635724db8476: Status 404 returned error can't find the container with id 597f6bf151d4a7a213d7a583b2240654bc91fdbf87042735e497635724db8476
Nov 03 20:17:47.777167 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.776953    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"]
Nov 03 20:17:47.779518 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:17:47.779487    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad11c25d_20ca_47d2_9bb0_03d4b4d3267b.slice/crio-55b91a33de0f87556f69afd4b42f02c028c559b4b3cd396f2cf701e323ea6bc7 WatchSource:0}: Error finding container 55b91a33de0f87556f69afd4b42f02c028c559b4b3cd396f2cf701e323ea6bc7: Status 404 returned error can't find the container with id 55b91a33de0f87556f69afd4b42f02c028c559b4b3cd396f2cf701e323ea6bc7
Nov 03 20:17:47.844273 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:47.844239    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"]
Nov 03 20:17:47.845634 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:17:47.845605    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc545c925_e23b_4155_8af0_edbdd2af211f.slice/crio-b70b68b3799d517111bf80a6f6e62bad64aa78b60953ba46fa4a696076b6f270 WatchSource:0}: Error finding container b70b68b3799d517111bf80a6f6e62bad64aa78b60953ba46fa4a696076b6f270: Status 404 returned error can't find the container with id b70b68b3799d517111bf80a6f6e62bad64aa78b60953ba46fa4a696076b6f270
Nov 03 20:17:48.032436 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.032342    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:48.734037 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.733997    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:48.748995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.748960    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd" event={"ID":"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b","Type":"ContainerStarted","Data":"1d90881d90fe88866a736dd06462ea1866711373fce7ab751165553690a75d59"}
Nov 03 20:17:48.749163 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.749027    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd" event={"ID":"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b","Type":"ContainerStarted","Data":"55b91a33de0f87556f69afd4b42f02c028c559b4b3cd396f2cf701e323ea6bc7"}
Nov 03 20:17:48.749262 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.749183    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:17:48.750686 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.750661    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb" event={"ID":"c545c925-e23b-4155-8af0-edbdd2af211f","Type":"ContainerStarted","Data":"95a5bdad64c450145fce20c40e27ce77477b051ed9f6ba4c3e25b1bdbd49b41c"}
Nov 03 20:17:48.750836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.750704    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb" event={"ID":"c545c925-e23b-4155-8af0-edbdd2af211f","Type":"ContainerStarted","Data":"b70b68b3799d517111bf80a6f6e62bad64aa78b60953ba46fa4a696076b6f270"}
Nov 03 20:17:48.750836 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.750768    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:17:48.752239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.752218    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd" event={"ID":"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50","Type":"ContainerStarted","Data":"afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8"}
Nov 03 20:17:48.752332 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.752258    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:48.752332 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.752273    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd" event={"ID":"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50","Type":"ContainerStarted","Data":"597f6bf151d4a7a213d7a583b2240654bc91fdbf87042735e497635724db8476"}
Nov 03 20:17:48.765777 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.765733    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd" podStartSLOduration=1.765716046 podStartE2EDuration="1.765716046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:48.763775514 +0000 UTC m=+2782.033898809" watchObservedRunningTime="2026-11-03 20:17:48.765716046 +0000 UTC m=+2782.035839342"
Nov 03 20:17:48.781648 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.781599    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd" podStartSLOduration=1.266688346 podStartE2EDuration="1.781585837s" totalImagesPullingTime="514.897491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:48.780445938 +0000 UTC m=+2782.050569233" watchObservedRunningTime="2026-11-03 20:17:48.781585837 +0000 UTC m=+2782.051709133"
Nov 03 20:17:48.796512 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:48.796469    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb" podStartSLOduration=1.796457022 podStartE2EDuration="1.796457022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:48.794033288 +0000 UTC m=+2782.064156561" watchObservedRunningTime="2026-11-03 20:17:48.796457022 +0000 UTC m=+2782.066580316"
Nov 03 20:17:49.426763 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:49.426720    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:50.325323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:50.325282    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:50.586090 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:17:50.585993    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:17:51.049201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:51.046157    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:51.835233 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:51.835167    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:52.711355 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:52.711313    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:53.540199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:53.536345    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:54.423674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:54.423632    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:54.765580 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:54.765496    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:17:55.229251 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:55.229215    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:56.102238 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:56.102202    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:56.828067 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:56.828026    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:57.526393 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:57.526352    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:58.302540 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:58.302498    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:59.014661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.014612    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:59.719390 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.719339    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:59.725640 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.725608    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Nov 03 20:17:59.725870 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.725683    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Nov 03 20:17:59.763719 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.763686    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:17:59.763921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.763852    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:17:59.819754 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.819717    4531 generic.go:356] "Generic (PLEG): container finished" podID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerID="c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864" exitCode=0
Nov 03 20:17:59.819946 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.819806    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" event={"ID":"b2948b4d-ffdd-49ca-84b5-1017eaef163a","Type":"ContainerDied","Data":"c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864"}
Nov 03 20:17:59.820360 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.820337    4531 scope.go:122] "RemoveContainer" containerID="c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864"
Nov 03 20:17:59.821858 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.821820    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:59.821978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.821930    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:59.822053 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:17:59.821981    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:18:00.557194 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:00.556285    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:00.827813 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:00.827727    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" event={"ID":"b2948b4d-ffdd-49ca-84b5-1017eaef163a","Type":"ContainerStarted","Data":"00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a"}
Nov 03 20:18:00.961988 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:00.961949    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:01.355161 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:01.355077    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:01.511532 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:01.511492    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:01.687153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:01.687116    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:18:01.687613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:01.687463    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Nov 03 20:18:01.687613 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:01.687512    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Nov 03 20:18:01.756133 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:01.756093    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:02.405858 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.405813    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"]
Nov 03 20:18:02.408882 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.408857    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.419230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.417479    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-794e97\"/\"kube-root-ca.crt\""
Nov 03 20:18:02.419230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.417554    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-794e97\"/\"openshift-service-ca.crt\""
Nov 03 20:18:02.425296 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.425267    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-794e97\"/\"simulator-single-main-794e97-kserve-self-signed-certs\""
Nov 03 20:18:02.425495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.425378    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-794e97\"/\"default-dockercfg-9z8q2\""
Nov 03 20:18:02.425580 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.425389    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"]
Nov 03 20:18:02.464199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.460645    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:02.505685 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.505649    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tls-certs\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.505685 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.505692    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tmp-dir\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.505904 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.505807    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-dshm\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.505904 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.505844    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscdn\" (UniqueName: \"kubernetes.io/projected/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-kube-api-access-kscdn\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.505981 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.505914    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-home\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.506017 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.505975    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-model-cache\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.606919 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.606868    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tmp-dir\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.607141 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.606960    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-dshm\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.607141 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.606993    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscdn\" (UniqueName: \"kubernetes.io/projected/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-kube-api-access-kscdn\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.607141 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.607032    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-home\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.607141 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.607085    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-model-cache\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.607429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.607199    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tls-certs\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.607495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.607422    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tmp-dir\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.607687 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.607660    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-home\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.607868 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.607699    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-model-cache\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.609422 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.609400    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-dshm\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.609830 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.609809    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tls-certs\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.616213 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.616190    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscdn\" (UniqueName: \"kubernetes.io/projected/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-kube-api-access-kscdn\") pod \"simulator-single-main-794e97-kserve-6b66c496f7-6brmd\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") " pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:02.622841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.622815    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:02.686629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.686542    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Nov 03 20:18:02.686629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.686598    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Nov 03 20:18:02.726653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:02.726620    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:03.026780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:03.026750    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"]
Nov 03 20:18:03.028566 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:18:03.028535    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fd5bb0a_0c67_4886_9265_c68ec8c8b4ce.slice/crio-30419ea179a9cd7bb44dde049913a5d9660497199bc56c8b3898e47b9bec22cf WatchSource:0}: Error finding container 30419ea179a9cd7bb44dde049913a5d9660497199bc56c8b3898e47b9bec22cf: Status 404 returned error can't find the container with id 30419ea179a9cd7bb44dde049913a5d9660497199bc56c8b3898e47b9bec22cf
Nov 03 20:18:03.358289 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:03.348080    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:03.687444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:03.687400    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Nov 03 20:18:03.687640 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:03.687480    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Nov 03 20:18:03.874310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:03.874275    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd" event={"ID":"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce","Type":"ContainerStarted","Data":"8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044"}
Nov 03 20:18:03.874723 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:03.874319    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd" event={"ID":"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce","Type":"ContainerStarted","Data":"30419ea179a9cd7bb44dde049913a5d9660497199bc56c8b3898e47b9bec22cf"}
Nov 03 20:18:03.890545 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:03.890496    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd" podStartSLOduration=1.8904840040000002 podStartE2EDuration="1.890484004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:03.890390821 +0000 UTC m=+2797.160514118" watchObservedRunningTime="2026-11-03 20:18:03.890484004 +0000 UTC m=+2797.160607298"
Nov 03 20:18:04.349444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:04.349405    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:04.686877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:04.686841    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Nov 03 20:18:04.687079 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:04.686902    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Nov 03 20:18:04.727211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:04.727156    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:05.214850 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:05.214820    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:05.686714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:05.686680    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Nov 03 20:18:05.686920 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:05.686733    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Nov 03 20:18:06.066545 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:06.066434    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:06.691878 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:06.691846    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:18:06.692388 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:06.692206    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:18:06.692857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:06.692839    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:18:06.834612 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:06.834576    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:07.534615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:07.534575    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:08.212370 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:08.212333    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:09.035528 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:09.035485    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:09.724007 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:09.723928    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:10.422814 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:10.422769    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:11.318119 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:11.318077    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:12.029222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:12.026761    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:12.729045 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:12.728960    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:12.742577 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:12.742544    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:12.743586 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:12.743555    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:12.934230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:12.934200    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:18:13.544643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:13.544604    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:14.291480 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:14.291437    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:15.012298 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:15.012264    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:15.867303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:15.867263    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:16.503336 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:16.503297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:17.225528 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:17.225483    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:17.422697 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:17.422649    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:17.624400 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:17.624314    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:17.916612 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:17.916576    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:18.631606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:18.631318    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:19.421674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:19.421629    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:20.117273 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:20.117154    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:20.573426 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:18:20.573384    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:18:21.025149 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:21.025111    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:21.822911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:21.822831    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:22.625929 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:22.625889    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:23.405473 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:23.405435    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:24.146134 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:24.146105    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:24.819009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:24.818967    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:25.515570 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:25.515530    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:26.222719 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:26.222667    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:26.899800 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:26.899761    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:27.614415 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:27.614378    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:28.315193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:28.315138    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:29.013993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:29.013954    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:29.716372 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:29.716328    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:30.505164 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:30.505128    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:31.214623 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:31.214560    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:31.925704 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:31.925669    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:32.631163 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:32.631103    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:33.336911 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:33.336872    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:34.237056 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:34.237022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:34.923305 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:34.923269    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:35.605910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:35.605826    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:36.023239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:36.023199    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"]
Nov 03 20:18:36.027289 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:36.027253    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6r878"]
Nov 03 20:18:36.316924 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:36.316768    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:36.667771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:36.667737    4531 scope.go:122] "RemoveContainer" containerID="56470292fefbaf8a204433bd3978c9e1e78ee08ef28e1b5b58bd9b77755cbc52"
Nov 03 20:18:36.677408 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:36.677383    4531 scope.go:122] "RemoveContainer" containerID="58d9c1ea42392f07ac8d9449c5a01bbcd85ceb8c72b7fa167b48eb8e16ca2293"
Nov 03 20:18:36.686318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:36.686295    4531 scope.go:122] "RemoveContainer" containerID="857e6cadaf870b7b9125f2c76f166ba87e7a0b2e05a5958161f4d7655f6db543"
Nov 03 20:18:36.857877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:36.857845    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4c5055f-eced-432e-bbed-620e7fdcad28" path="/var/lib/kubelet/pods/c4c5055f-eced-432e-bbed-620e7fdcad28/volumes"
Nov 03 20:18:37.006074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:37.005967    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:37.730248 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:37.730203    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:38.506664 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:38.506627    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:39.310048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:39.309997    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:40.107750 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:40.107711    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:40.821632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:40.821598    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:41.520220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:41.518245    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:42.415447 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:42.415411    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:43.117382 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:43.117337    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:43.604900 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:43.604858    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:43.887487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:43.887443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:44.807940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:44.807764    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:45.538502 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:45.538457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:46.313744 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:46.313702    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:47.021015 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:47.020977    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:47.021282 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:47.021242    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"]
Nov 03 20:18:47.023584 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:47.023555    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jctzd"]
Nov 03 20:18:47.816496 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:47.816462    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:48.524407 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:48.524372    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:48.620457 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:48.620421    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"]
Nov 03 20:18:48.743500 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:48.743463    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:48.857878 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:48.857800    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2882008e-047a-41f5-8a1e-35490cb73074" path="/var/lib/kubelet/pods/2882008e-047a-41f5-8a1e-35490cb73074/volumes"
Nov 03 20:18:49.318578 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:49.318543    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:50.033380 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:50.033341    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:50.574601 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:18:50.574555    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:18:50.820459 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:50.820425    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:51.520378 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:51.520339    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:52.235728 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:52.235691    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:52.937813 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:52.937784    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:53.615322 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:53.615288    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:53.969144 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:53.969104    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"]
Nov 03 20:18:53.974496 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:53.974467    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"]
Nov 03 20:18:54.152151 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.152093    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:54.174831 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.174794    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"]
Nov 03 20:18:54.175121 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.175088    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd" podUID="7e6eeea2-b2b3-4a7a-b9a2-50f003198a50" containerName="maas-api" containerID="cri-o://afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8" gracePeriod=30
Nov 03 20:18:54.279078 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.279000    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:54.633001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.632973    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:18:54.782949 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.782911    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-maas-api-tls\") pod \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") "
Nov 03 20:18:54.783160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.782967    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-metrics-certs\") pod \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") "
Nov 03 20:18:54.783160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.782995    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-openshift-service-ca\") pod \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") "
Nov 03 20:18:54.783160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.783025    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-kube-api-access-9qhp8\" (UniqueName: \"kubernetes.io/projected/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-kube-api-access-9qhp8\") pod \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\" (UID: \"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50\") "
Nov 03 20:18:54.783471 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.783445    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-openshift-service-ca" pod "7e6eeea2-b2b3-4a7a-b9a2-50f003198a50" (UID: "7e6eeea2-b2b3-4a7a-b9a2-50f003198a50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:54.785343 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.785321    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-maas-api-tls" pod "7e6eeea2-b2b3-4a7a-b9a2-50f003198a50" (UID: "7e6eeea2-b2b3-4a7a-b9a2-50f003198a50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:54.785435 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.785370    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-metrics-certs" pod "7e6eeea2-b2b3-4a7a-b9a2-50f003198a50" (UID: "7e6eeea2-b2b3-4a7a-b9a2-50f003198a50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:54.785435 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.785374    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-kube-api-access-9qhp8" pod "7e6eeea2-b2b3-4a7a-b9a2-50f003198a50" (UID: "7e6eeea2-b2b3-4a7a-b9a2-50f003198a50"). InnerVolumeSpecName "kube-api-access-9qhp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:54.884638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.884602    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:54.884638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.884633    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:54.884821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.884643    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:54.884821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:54.884654    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qhp8\" (UniqueName: \"kubernetes.io/projected/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50-kube-api-access-9qhp8\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:55.068359 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.068317    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:55.161479 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.161435    4531 generic.go:356] "Generic (PLEG): container finished" podID="7e6eeea2-b2b3-4a7a-b9a2-50f003198a50" containerID="afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8" exitCode=0
Nov 03 20:18:55.161675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.161504    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"
Nov 03 20:18:55.161675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.161519    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd" event={"ID":"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50","Type":"ContainerDied","Data":"afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8"}
Nov 03 20:18:55.161675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.161582    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd" event={"ID":"7e6eeea2-b2b3-4a7a-b9a2-50f003198a50","Type":"ContainerDied","Data":"597f6bf151d4a7a213d7a583b2240654bc91fdbf87042735e497635724db8476"}
Nov 03 20:18:55.161675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.161608    4531 scope.go:122] "RemoveContainer" containerID="afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8"
Nov 03 20:18:55.171100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.171078    4531 scope.go:122] "RemoveContainer" containerID="afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8"
Nov 03 20:18:55.171384 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:18:55.171360    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8\": container with ID starting with afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8 not found: ID does not exist" containerID="afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8"
Nov 03 20:18:55.171502 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.171391    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8"} err="failed to get container status \"afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8\": rpc error: code = NotFound desc = could not find container \"afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8\": container with ID starting with afdfdc6ade2da8600357be15e0ec3f7b796e5fab69ed6a5716d10dc7a38aeca8 not found: ID does not exist"
Nov 03 20:18:55.176872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.176849    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"]
Nov 03 20:18:55.182888 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.182864    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-794e97-755554cf5c-vn5hd"]
Nov 03 20:18:55.202610 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.202582    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:55.579878 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:55.579793    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:56.857600 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:56.857568    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e6eeea2-b2b3-4a7a-b9a2-50f003198a50" path="/var/lib/kubelet/pods/7e6eeea2-b2b3-4a7a-b9a2-50f003198a50/volumes"
Nov 03 20:18:58.969867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:58.969822    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb" podUID="c545c925-e23b-4155-8af0-edbdd2af211f" containerName="payload-processing" containerID="cri-o://95a5bdad64c450145fce20c40e27ce77477b051ed9f6ba4c3e25b1bdbd49b41c" gracePeriod=25
Nov 03 20:18:58.975009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:58.974978    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd" podUID="ad11c25d-20ca-47d2-9bb0-03d4b4d3267b" containerName="payload-pre-processing" containerID="cri-o://1d90881d90fe88866a736dd06462ea1866711373fce7ab751165553690a75d59" gracePeriod=25
Nov 03 20:18:59.184547 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.184510    4531 generic.go:356] "Generic (PLEG): container finished" podID="ad11c25d-20ca-47d2-9bb0-03d4b4d3267b" containerID="1d90881d90fe88866a736dd06462ea1866711373fce7ab751165553690a75d59" exitCode=0
Nov 03 20:18:59.184750 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.184581    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd" event={"ID":"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b","Type":"ContainerDied","Data":"1d90881d90fe88866a736dd06462ea1866711373fce7ab751165553690a75d59"}
Nov 03 20:18:59.186621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.186593    4531 generic.go:356] "Generic (PLEG): container finished" podID="c545c925-e23b-4155-8af0-edbdd2af211f" containerID="95a5bdad64c450145fce20c40e27ce77477b051ed9f6ba4c3e25b1bdbd49b41c" exitCode=0
Nov 03 20:18:59.186756 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.186624    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb" event={"ID":"c545c925-e23b-4155-8af0-edbdd2af211f","Type":"ContainerDied","Data":"95a5bdad64c450145fce20c40e27ce77477b051ed9f6ba4c3e25b1bdbd49b41c"}
Nov 03 20:18:59.225341 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.225269    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:18:59.236510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.236483    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c545c925-e23b-4155-8af0-edbdd2af211f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c545c925-e23b-4155-8af0-edbdd2af211f-plugins-config-volume\") pod \"c545c925-e23b-4155-8af0-edbdd2af211f\" (UID: \"c545c925-e23b-4155-8af0-edbdd2af211f\") "
Nov 03 20:18:59.236686 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.236668    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c545c925-e23b-4155-8af0-edbdd2af211f-kube-api-access-nkt25\" (UniqueName: \"kubernetes.io/projected/c545c925-e23b-4155-8af0-edbdd2af211f-kube-api-access-nkt25\") pod \"c545c925-e23b-4155-8af0-edbdd2af211f\" (UID: \"c545c925-e23b-4155-8af0-edbdd2af211f\") "
Nov 03 20:18:59.236881 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.236859    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c545c925-e23b-4155-8af0-edbdd2af211f-plugins-config-volume" pod "c545c925-e23b-4155-8af0-edbdd2af211f" (UID: "c545c925-e23b-4155-8af0-edbdd2af211f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:59.237048 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.237029    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c545c925-e23b-4155-8af0-edbdd2af211f-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:59.238834 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.238808    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c545c925-e23b-4155-8af0-edbdd2af211f-kube-api-access-nkt25" pod "c545c925-e23b-4155-8af0-edbdd2af211f" (UID: "c545c925-e23b-4155-8af0-edbdd2af211f"). InnerVolumeSpecName "kube-api-access-nkt25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:59.258953 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.258924    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:18:59.337616 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.337579    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-kube-api-access-ncjqx\" (UniqueName: \"kubernetes.io/projected/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-kube-api-access-ncjqx\") pod \"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b\" (UID: \"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b\") "
Nov 03 20:18:59.337838 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.337635    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-plugins-config-volume\") pod \"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b\" (UID: \"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b\") "
Nov 03 20:18:59.337838 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.337836    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkt25\" (UniqueName: \"kubernetes.io/projected/c545c925-e23b-4155-8af0-edbdd2af211f-kube-api-access-nkt25\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:59.338044 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.338012    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-plugins-config-volume" pod "ad11c25d-20ca-47d2-9bb0-03d4b4d3267b" (UID: "ad11c25d-20ca-47d2-9bb0-03d4b4d3267b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:59.339800 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.339778    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-kube-api-access-ncjqx" pod "ad11c25d-20ca-47d2-9bb0-03d4b4d3267b" (UID: "ad11c25d-20ca-47d2-9bb0-03d4b4d3267b"). InnerVolumeSpecName "kube-api-access-ncjqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:59.438755 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.438709    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncjqx\" (UniqueName: \"kubernetes.io/projected/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-kube-api-access-ncjqx\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:59.438755 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:18:59.438744    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:00.193256 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.193214    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd" event={"ID":"ad11c25d-20ca-47d2-9bb0-03d4b4d3267b","Type":"ContainerDied","Data":"55b91a33de0f87556f69afd4b42f02c028c559b4b3cd396f2cf701e323ea6bc7"}
Nov 03 20:19:00.193775 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.193236    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"
Nov 03 20:19:00.193775 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.193283    4531 scope.go:122] "RemoveContainer" containerID="1d90881d90fe88866a736dd06462ea1866711373fce7ab751165553690a75d59"
Nov 03 20:19:00.194879 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.194859    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"
Nov 03 20:19:00.194977 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.194903    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb" event={"ID":"c545c925-e23b-4155-8af0-edbdd2af211f","Type":"ContainerDied","Data":"b70b68b3799d517111bf80a6f6e62bad64aa78b60953ba46fa4a696076b6f270"}
Nov 03 20:19:00.206455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.206435    4531 scope.go:122] "RemoveContainer" containerID="95a5bdad64c450145fce20c40e27ce77477b051ed9f6ba4c3e25b1bdbd49b41c"
Nov 03 20:19:00.219279 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.219253    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"]
Nov 03 20:19:00.221768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.221744    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-794e97-7b68cbtvpd"]
Nov 03 20:19:00.229287 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.229262    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"]
Nov 03 20:19:00.232853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.232795    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-794e97-7f5657f55lkvmb"]
Nov 03 20:19:00.861793 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.861754    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad11c25d-20ca-47d2-9bb0-03d4b4d3267b" path="/var/lib/kubelet/pods/ad11c25d-20ca-47d2-9bb0-03d4b4d3267b/volumes"
Nov 03 20:19:00.862485 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:00.862467    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c545c925-e23b-4155-8af0-edbdd2af211f" path="/var/lib/kubelet/pods/c545c925-e23b-4155-8af0-edbdd2af211f/volumes"
Nov 03 20:19:01.309417 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:01.309385    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:01.454466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:01.454426    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:01.555716 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:01.555648    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"]
Nov 03 20:19:01.556274 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:01.556241    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy" containerID="cri-o://00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a" gracePeriod=30
Nov 03 20:19:01.639559 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:01.639511    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:03.641612 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.641556    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd" podUID="4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" containerName="main" containerID="cri-o://8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044" gracePeriod=45
Nov 03 20:19:03.889675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.889649    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:19:03.987811 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.987720    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-kube-api-access-kscdn\" (UniqueName: \"kubernetes.io/projected/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-kube-api-access-kscdn\") pod \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") "
Nov 03 20:19:03.987811 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.987765    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-home\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-home\") pod \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") "
Nov 03 20:19:03.987811 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.987799    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-dshm\") pod \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") "
Nov 03 20:19:03.988518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.987816    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tmp-dir\") pod \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") "
Nov 03 20:19:03.988518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.987888    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tls-certs\") pod \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") "
Nov 03 20:19:03.988518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.987959    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-model-cache\") pod \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\" (UID: \"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce\") "
Nov 03 20:19:03.988518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.988392    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-home" pod "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" (UID: "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:03.988518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.988484    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-model-cache" pod "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" (UID: "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:03.988518 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.988502    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tmp-dir" pod "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" (UID: "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:03.990646 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.990603    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tls-certs" pod "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" (UID: "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:03.990646 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.990613    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-dshm" pod "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" (UID: "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:03.990828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:03.990728    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-kube-api-access-kscdn" pod "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" (UID: "4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce"). InnerVolumeSpecName "kube-api-access-kscdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:04.089649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.089609    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscdn\" (UniqueName: \"kubernetes.io/projected/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-kube-api-access-kscdn\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:04.089649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.089644    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:04.089649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.089655    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:04.089919 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.089663    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:04.089919 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.089672    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:04.089919 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.089680    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:04.219018 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.218983    4531 generic.go:356] "Generic (PLEG): container finished" podID="4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" containerID="8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044" exitCode=0
Nov 03 20:19:04.219227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.219031    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"
Nov 03 20:19:04.219227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.219078    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd" event={"ID":"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce","Type":"ContainerDied","Data":"8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044"}
Nov 03 20:19:04.219227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.219119    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd" event={"ID":"4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce","Type":"ContainerDied","Data":"30419ea179a9cd7bb44dde049913a5d9660497199bc56c8b3898e47b9bec22cf"}
Nov 03 20:19:04.219227 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.219138    4531 scope.go:122] "RemoveContainer" containerID="8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044"
Nov 03 20:19:04.228988 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.228969    4531 scope.go:122] "RemoveContainer" containerID="8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044"
Nov 03 20:19:04.229308 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:19:04.229288    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044\": container with ID starting with 8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044 not found: ID does not exist" containerID="8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044"
Nov 03 20:19:04.229362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.229318    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044"} err="failed to get container status \"8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044\": rpc error: code = NotFound desc = could not find container \"8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044\": container with ID starting with 8ad1b642ad9d062f593e807c88d41f1d9aea6528abb8fbc76aac6cfa5dd2f044 not found: ID does not exist"
Nov 03 20:19:04.240551 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.240500    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"]
Nov 03 20:19:04.244219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.244195    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-794e97/simulator-single-main-794e97-kserve-6b66c496f7-6brmd"]
Nov 03 20:19:04.857127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:04.857092    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" path="/var/lib/kubelet/pods/4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce/volumes"
Nov 03 20:19:05.539289 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.539244    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"]
Nov 03 20:19:05.539912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.539890    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ad11c25d-20ca-47d2-9bb0-03d4b4d3267b" containerName="payload-pre-processing"
Nov 03 20:19:05.539978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.539916    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad11c25d-20ca-47d2-9bb0-03d4b4d3267b" containerName="payload-pre-processing"
Nov 03 20:19:05.539978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.539938    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7e6eeea2-b2b3-4a7a-b9a2-50f003198a50" containerName="maas-api"
Nov 03 20:19:05.539978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.539946    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6eeea2-b2b3-4a7a-b9a2-50f003198a50" containerName="maas-api"
Nov 03 20:19:05.540099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.539979    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" containerName="main"
Nov 03 20:19:05.540099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.539987    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" containerName="main"
Nov 03 20:19:05.540099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.540024    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c545c925-e23b-4155-8af0-edbdd2af211f" containerName="payload-processing"
Nov 03 20:19:05.540099 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.540031    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c545c925-e23b-4155-8af0-edbdd2af211f" containerName="payload-processing"
Nov 03 20:19:05.540259 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.540129    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6eeea2-b2b3-4a7a-b9a2-50f003198a50" containerName="maas-api"
Nov 03 20:19:05.540259 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.540146    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd5bb0a-0c67-4886-9265-c68ec8c8b4ce" containerName="main"
Nov 03 20:19:05.540259 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.540161    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c545c925-e23b-4155-8af0-edbdd2af211f" containerName="payload-processing"
Nov 03 20:19:05.540259 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.540188    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad11c25d-20ca-47d2-9bb0-03d4b4d3267b" containerName="payload-pre-processing"
Nov 03 20:19:05.545008 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.544981    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.550915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.550886    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a0ada1a8-kserve-self-signed-certs\""
Nov 03 20:19:05.551329 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.551307    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"]
Nov 03 20:19:05.604054 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.604020    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzjhz\" (UniqueName: \"kubernetes.io/projected/2f1560ba-749d-4336-a0a3-8fff53d66950-kube-api-access-bzjhz\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.604291 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.604082    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-tmp-dir\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.604291 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.604223    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-home\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.604291 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.604263    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1560ba-749d-4336-a0a3-8fff53d66950-tls-certs\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.604478 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.604354    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-model-cache\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.604478 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.604412    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-dshm\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.635547 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.635515    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:05.705105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.705071    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjhz\" (UniqueName: \"kubernetes.io/projected/2f1560ba-749d-4336-a0a3-8fff53d66950-kube-api-access-bzjhz\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.705349 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.705135    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-tmp-dir\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.705349 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.705195    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-home\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.705349 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.705247    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1560ba-749d-4336-a0a3-8fff53d66950-tls-certs\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.705349 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.705280    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-model-cache\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.705594 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.705393    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-dshm\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.705666 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.705592    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-tmp-dir\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.705666 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.705626    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-home\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.705762 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.705685    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-model-cache\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.707630 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.707608    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-dshm\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.707916 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.707894    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1560ba-749d-4336-a0a3-8fff53d66950-tls-certs\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.711866 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.711843    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzjhz\" (UniqueName: \"kubernetes.io/projected/2f1560ba-749d-4336-a0a3-8fff53d66950-kube-api-access-bzjhz\") pod \"e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") " pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:05.860878 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:05.860790    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:06.135093 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.135037    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"]
Nov 03 20:19:06.137581 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:19:06.137542    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f1560ba_749d_4336_a0a3_8fff53d66950.slice/crio-93d53154d40293cadbf132a928eb03d1d22824c3bf550da63d589cd2923f46e4 WatchSource:0}: Error finding container 93d53154d40293cadbf132a928eb03d1d22824c3bf550da63d589cd2923f46e4: Status 404 returned error can't find the container with id 93d53154d40293cadbf132a928eb03d1d22824c3bf550da63d589cd2923f46e4
Nov 03 20:19:06.232437 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.232388    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg" event={"ID":"2f1560ba-749d-4336-a0a3-8fff53d66950","Type":"ContainerStarted","Data":"c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e"}
Nov 03 20:19:06.232437 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.232443    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg" event={"ID":"2f1560ba-749d-4336-a0a3-8fff53d66950","Type":"ContainerStarted","Data":"93d53154d40293cadbf132a928eb03d1d22824c3bf550da63d589cd2923f46e4"}
Nov 03 20:19:06.247754 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.247701    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg" podStartSLOduration=1.247686752 podStartE2EDuration="1.247686752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:06.246348997 +0000 UTC m=+2859.516472312" watchObservedRunningTime="2026-11-03 20:19:06.247686752 +0000 UTC m=+2859.517810047"
Nov 03 20:19:06.693477 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.693441    4531 patch_prober.go:29] interesting pod/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body=
Nov 03 20:19:06.693635 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.693505    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused"
Nov 03 20:19:06.806904 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.806874    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:19:06.861748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.861674    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:06.917958 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.917925    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-podinfo\") pod \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") "
Nov 03 20:19:06.918200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.917998    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-kube-api-access-2sj6h\" (UniqueName: \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-kube-api-access-2sj6h\") pod \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") "
Nov 03 20:19:06.918200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918066    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-token\" (UniqueName: \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-token\") pod \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") "
Nov 03 20:19:06.918200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918121    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istiod-ca-cert\") pod \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") "
Nov 03 20:19:06.918200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918198    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-envoy\") pod \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") "
Nov 03 20:19:06.918466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918246    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-data\") pod \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") "
Nov 03 20:19:06.918466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918287    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-credential-socket\") pod \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") "
Nov 03 20:19:06.918466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918319    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-socket\") pod \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") "
Nov 03 20:19:06.918466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918347    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-certs\") pod \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\" (UID: \"b2948b4d-ffdd-49ca-84b5-1017eaef163a\") "
Nov 03 20:19:06.918756 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918672    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istiod-ca-cert" pod "b2948b4d-ffdd-49ca-84b5-1017eaef163a" (UID: "b2948b4d-ffdd-49ca-84b5-1017eaef163a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:06.918756 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918693    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-credential-socket" pod "b2948b4d-ffdd-49ca-84b5-1017eaef163a" (UID: "b2948b4d-ffdd-49ca-84b5-1017eaef163a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:06.918875 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918756    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-data" pod "b2948b4d-ffdd-49ca-84b5-1017eaef163a" (UID: "b2948b4d-ffdd-49ca-84b5-1017eaef163a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:06.918875 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918848    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-socket" pod "b2948b4d-ffdd-49ca-84b5-1017eaef163a" (UID: "b2948b4d-ffdd-49ca-84b5-1017eaef163a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:06.918992 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.918925    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-certs" pod "b2948b4d-ffdd-49ca-84b5-1017eaef163a" (UID: "b2948b4d-ffdd-49ca-84b5-1017eaef163a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:06.920488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.920461    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-podinfo" pod "b2948b4d-ffdd-49ca-84b5-1017eaef163a" (UID: "b2948b4d-ffdd-49ca-84b5-1017eaef163a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:06.920488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.920473    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-kube-api-access-2sj6h" pod "b2948b4d-ffdd-49ca-84b5-1017eaef163a" (UID: "b2948b4d-ffdd-49ca-84b5-1017eaef163a"). InnerVolumeSpecName "kube-api-access-2sj6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:06.920802 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.920777    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-token" pod "b2948b4d-ffdd-49ca-84b5-1017eaef163a" (UID: "b2948b4d-ffdd-49ca-84b5-1017eaef163a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:06.920896 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:06.920809    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-envoy" pod "b2948b4d-ffdd-49ca-84b5-1017eaef163a" (UID: "b2948b4d-ffdd-49ca-84b5-1017eaef163a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:07.019527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.019495    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:07.019527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.019524    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sj6h\" (UniqueName: \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-kube-api-access-2sj6h\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:07.019527 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.019535    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:07.019789 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.019543    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:07.019789 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.019552    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:07.019789 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.019560    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:07.019789 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.019568    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:07.019789 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.019576    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:07.019789 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.019585    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2948b4d-ffdd-49ca-84b5-1017eaef163a-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:07.239269 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.239230    4531 generic.go:356] "Generic (PLEG): container finished" podID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerID="00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a" exitCode=0
Nov 03 20:19:07.239476 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.239301    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"
Nov 03 20:19:07.239476 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.239325    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" event={"ID":"b2948b4d-ffdd-49ca-84b5-1017eaef163a","Type":"ContainerDied","Data":"00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a"}
Nov 03 20:19:07.239476 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.239386    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8" event={"ID":"b2948b4d-ffdd-49ca-84b5-1017eaef163a","Type":"ContainerDied","Data":"49364422d82a9b680f1f22d4cb6a0a3269f7e95d9b1081669d79a12006b8a8df"}
Nov 03 20:19:07.239476 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.239411    4531 scope.go:122] "RemoveContainer" containerID="00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a"
Nov 03 20:19:07.249605 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.249588    4531 scope.go:122] "RemoveContainer" containerID="c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864"
Nov 03 20:19:07.258168 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.258148    4531 scope.go:122] "RemoveContainer" containerID="00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a"
Nov 03 20:19:07.258451 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:19:07.258424    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a\": container with ID starting with 00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a not found: ID does not exist" containerID="00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a"
Nov 03 20:19:07.258549 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.258463    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a"} err="failed to get container status \"00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a\": rpc error: code = NotFound desc = could not find container \"00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a\": container with ID starting with 00ae0bdc075ad89741e322fc7b6de4aa15b5bf407693994b9dbe2e4b0621221a not found: ID does not exist"
Nov 03 20:19:07.258549 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.258487    4531 scope.go:122] "RemoveContainer" containerID="c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864"
Nov 03 20:19:07.258723 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:19:07.258705    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864\": container with ID starting with c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864 not found: ID does not exist" containerID="c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864"
Nov 03 20:19:07.258764 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.258731    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864"} err="failed to get container status \"c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864\": rpc error: code = NotFound desc = could not find container \"c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864\": container with ID starting with c6405448cb58c2141c91b102cfb88372d13482bfd155c20b8b571bb1357b3864 not found: ID does not exist"
Nov 03 20:19:07.272713 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.272685    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"]
Nov 03 20:19:07.281935 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:07.281914    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-794e97-openshift-default-6bd758c985ggft8"]
Nov 03 20:19:08.894198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:08.881431    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" path="/var/lib/kubelet/pods/b2948b4d-ffdd-49ca-84b5-1017eaef163a/volumes"
Nov 03 20:19:15.861654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:15.861615    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:15.874674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:15.874644    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:16.319948 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:16.319918    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:19:17.667629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:17.667596    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:20.572869 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:19:20.572826    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:19:24.226109 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.226074    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"]
Nov 03 20:19:24.226649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.226530    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy"
Nov 03 20:19:24.226649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.226544    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy"
Nov 03 20:19:24.226649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.226565    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy"
Nov 03 20:19:24.226649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.226572    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy"
Nov 03 20:19:24.226649 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.226651    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy"
Nov 03 20:19:24.226973 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.226665    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2948b4d-ffdd-49ca-84b5-1017eaef163a" containerName="istio-proxy"
Nov 03 20:19:24.231531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.231504    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.234799 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.234521    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a0ada1a8-kserve-self-signed-certs\""
Nov 03 20:19:24.237679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.237647    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"]
Nov 03 20:19:24.324791 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.324750    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:24.332652 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.332618    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tls-certs\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.332801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.332684    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tmp-dir\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.332841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.332811    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-home\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.332884 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.332844    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dgl\" (UniqueName: \"kubernetes.io/projected/19f810dc-b163-4c59-b17a-8b11e5b8fef0-kube-api-access-w6dgl\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.332884 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.332872    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-dshm\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.333008 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.332990    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-model-cache\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.434074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.434020    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-home\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.434074 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.434080    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dgl\" (UniqueName: \"kubernetes.io/projected/19f810dc-b163-4c59-b17a-8b11e5b8fef0-kube-api-access-w6dgl\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.434396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.434107    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-dshm\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.434396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.434134    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-model-cache\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.434396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.434202    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tls-certs\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.434396 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.434240    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tmp-dir\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.434629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.434569    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-home\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.434629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.434603    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-model-cache\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.434701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.434656    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tmp-dir\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.436554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.436528    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-dshm\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.436982 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.436960    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tls-certs\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.448497 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.448472    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dgl\" (UniqueName: \"kubernetes.io/projected/19f810dc-b163-4c59-b17a-8b11e5b8fef0-kube-api-access-w6dgl\") pod \"e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") " pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.550541 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.550439    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:24.826858 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:24.826831    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"]
Nov 03 20:19:24.828915 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:19:24.828883    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19f810dc_b163_4c59_b17a_8b11e5b8fef0.slice/crio-371acbd45015d2eb7efc2ab3f415ff873cb79baf3ea7e85f8f9d2eb8fa7110d7 WatchSource:0}: Error finding container 371acbd45015d2eb7efc2ab3f415ff873cb79baf3ea7e85f8f9d2eb8fa7110d7: Status 404 returned error can't find the container with id 371acbd45015d2eb7efc2ab3f415ff873cb79baf3ea7e85f8f9d2eb8fa7110d7
Nov 03 20:19:25.355571 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:25.355531    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n" event={"ID":"19f810dc-b163-4c59-b17a-8b11e5b8fef0","Type":"ContainerStarted","Data":"62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0"}
Nov 03 20:19:25.355571 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:25.355578    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n" event={"ID":"19f810dc-b163-4c59-b17a-8b11e5b8fef0","Type":"ContainerStarted","Data":"371acbd45015d2eb7efc2ab3f415ff873cb79baf3ea7e85f8f9d2eb8fa7110d7"}
Nov 03 20:19:25.372391 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:25.372338    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n" podStartSLOduration=1.372320594 podStartE2EDuration="1.372320594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:25.371268341 +0000 UTC m=+2878.641391637" watchObservedRunningTime="2026-11-03 20:19:25.372320594 +0000 UTC m=+2878.642443889"
Nov 03 20:19:25.551335 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:25.551296    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:26.019320 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:26.019267    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:26.157236 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:26.157166    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:29.729569 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:29.729533    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:29.856961 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:29.856925    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:31.814340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:31.814304    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:34.550804 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:34.550769    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:34.563421 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:34.563391    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:35.456294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:35.456256    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:19:36.719780 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:36.719748    4531 scope.go:122] "RemoveContainer" containerID="b86bbe2de73132d3ada9b24d00f6a802ea72632d3587ced33333e8cac2738eab"
Nov 03 20:19:36.728821 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:36.728799    4531 scope.go:122] "RemoveContainer" containerID="17b8c1e5c0b32ba33a970b8d88b72da23fbbc3af4e951a3c25b71fd47f046fe2"
Nov 03 20:19:36.736833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:36.736812    4531 scope.go:122] "RemoveContainer" containerID="acd8eb9dcf8976c65c105742a7f1eb9afe69099e714110e2350d3323c7f3c632"
Nov 03 20:19:36.744571 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:36.744550    4531 scope.go:122] "RemoveContainer" containerID="f782e95f242d19a7b7d6f7b6f62798eb432abb82ff00c355d97b0c415cb1f6ce"
Nov 03 20:19:38.917391 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:38.917352    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"]
Nov 03 20:19:38.922831 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:38.922810    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:38.926072 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:38.926036    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:19:38.926247 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:38.926137    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:19:38.927200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:38.927159    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"]
Nov 03 20:19:38.937934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:38.937143    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:19:38.937934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:38.937229    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:19:39.017536 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:39.017496    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szpsb\" (UniqueName: \"kubernetes.io/projected/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-kube-api-access-szpsb\") pod \"maas-api-revoke-sub-452b6d7b-4pw88\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:39.017536 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:39.017544    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4pw88\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:39.017791 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:39.017568    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4pw88\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:39.118948 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:39.118912    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4pw88\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:39.118948 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:39.118949    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4pw88\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:39.119266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:39.119145    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szpsb\" (UniqueName: \"kubernetes.io/projected/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-kube-api-access-szpsb\") pod \"maas-api-revoke-sub-452b6d7b-4pw88\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:39.119730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:39.119700    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4pw88\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:39.121379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:39.121361    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4pw88\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:39.126495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:39.126467    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szpsb\" (UniqueName: \"kubernetes.io/projected/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-kube-api-access-szpsb\") pod \"maas-api-revoke-sub-452b6d7b-4pw88\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:39.238506 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:39.238399    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:39.508761 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:39.508730    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"]
Nov 03 20:19:39.510717 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:19:39.510680    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73b4a5b_6f04_4b6e_9f1d_149a9aef4388.slice/crio-a3aed04b414a5e7d8d0295ce41b8889181ea54f91f54e2aa644069ebaffd3b67 WatchSource:0}: Error finding container a3aed04b414a5e7d8d0295ce41b8889181ea54f91f54e2aa644069ebaffd3b67: Status 404 returned error can't find the container with id a3aed04b414a5e7d8d0295ce41b8889181ea54f91f54e2aa644069ebaffd3b67
Nov 03 20:19:40.476184 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:40.476135    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88" event={"ID":"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388","Type":"ContainerStarted","Data":"200f60c505f43f2e6b3ee831ff2d477f695c736052ab6b8ae21e86520f6871f1"}
Nov 03 20:19:40.476730 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:40.476232    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88" event={"ID":"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388","Type":"ContainerStarted","Data":"a3aed04b414a5e7d8d0295ce41b8889181ea54f91f54e2aa644069ebaffd3b67"}
Nov 03 20:19:40.491634 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:40.491566    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88" podStartSLOduration=2.491548884 podStartE2EDuration="2.491548884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:40.490201957 +0000 UTC m=+2893.760325254" watchObservedRunningTime="2026-11-03 20:19:40.491548884 +0000 UTC m=+2893.761672182"
Nov 03 20:19:41.500001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:41.499960    4531 generic.go:356] "Generic (PLEG): container finished" podID="e73b4a5b-6f04-4b6e-9f1d-149a9aef4388" containerID="200f60c505f43f2e6b3ee831ff2d477f695c736052ab6b8ae21e86520f6871f1" exitCode=0
Nov 03 20:19:41.500671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:41.500638    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88" event={"ID":"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388","Type":"ContainerDied","Data":"200f60c505f43f2e6b3ee831ff2d477f695c736052ab6b8ae21e86520f6871f1"}
Nov 03 20:19:43.215198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.215138    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:43.303531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.303466    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-kube-api-access-szpsb\" (UniqueName: \"kubernetes.io/projected/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-kube-api-access-szpsb\") pod \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") "
Nov 03 20:19:43.303967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.303920    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-openshift-ca\") pod \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") "
Nov 03 20:19:43.304325 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.304303    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-xks-ca\") pod \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\" (UID: \"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388\") "
Nov 03 20:19:43.309670 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.309636    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-openshift-ca" pod "e73b4a5b-6f04-4b6e-9f1d-149a9aef4388" (UID: "e73b4a5b-6f04-4b6e-9f1d-149a9aef4388"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:43.313658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.313632    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-xks-ca" pod "e73b4a5b-6f04-4b6e-9f1d-149a9aef4388" (UID: "e73b4a5b-6f04-4b6e-9f1d-149a9aef4388"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:43.315354 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.315325    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-kube-api-access-szpsb" pod "e73b4a5b-6f04-4b6e-9f1d-149a9aef4388" (UID: "e73b4a5b-6f04-4b6e-9f1d-149a9aef4388"). InnerVolumeSpecName "kube-api-access-szpsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:43.406374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.406331    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:43.406374 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.406372    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szpsb\" (UniqueName: \"kubernetes.io/projected/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-kube-api-access-szpsb\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:43.406609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.406383    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:43.532515 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.532482    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"
Nov 03 20:19:43.532701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.532513    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88" event={"ID":"e73b4a5b-6f04-4b6e-9f1d-149a9aef4388","Type":"ContainerDied","Data":"a3aed04b414a5e7d8d0295ce41b8889181ea54f91f54e2aa644069ebaffd3b67"}
Nov 03 20:19:43.532701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:43.532563    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3aed04b414a5e7d8d0295ce41b8889181ea54f91f54e2aa644069ebaffd3b67"
Nov 03 20:19:49.720270 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:49.720227    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"]
Nov 03 20:19:49.814887 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:49.814855    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:50.573813 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:19:50.573776    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:19:51.163681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:51.163644    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"]
Nov 03 20:19:51.253900 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:51.253863    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:52.592822 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.592783    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:52.620874 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.620831    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"]
Nov 03 20:19:52.621962 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.621919    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e73b4a5b-6f04-4b6e-9f1d-149a9aef4388" containerName="revoke-keys"
Nov 03 20:19:52.622163 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.622146    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73b4a5b-6f04-4b6e-9f1d-149a9aef4388" containerName="revoke-keys"
Nov 03 20:19:52.634083 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.634055    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73b4a5b-6f04-4b6e-9f1d-149a9aef4388" containerName="revoke-keys"
Nov 03 20:19:52.642650 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.642620    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"]
Nov 03 20:19:52.642989 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.642968    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:52.680722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.680523    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:19:52.680722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.680676    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:19:52.705916 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.705785    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-n586q\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:52.705916 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.705826    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:19:52.705916 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.705864    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-n586q\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:52.705916 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.705872    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:19:52.706339 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.706079    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgkh9\" (UniqueName: \"kubernetes.io/projected/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-kube-api-access-bgkh9\") pod \"maas-api-revoke-sub-3e31a506-n586q\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:52.823223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.822482    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkh9\" (UniqueName: \"kubernetes.io/projected/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-kube-api-access-bgkh9\") pod \"maas-api-revoke-sub-3e31a506-n586q\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:52.823223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.822534    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-n586q\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:52.823223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.822599    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-n586q\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:52.844688 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.844607    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-n586q\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:52.886195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.853258    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-n586q\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:52.900254 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:52.888603    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgkh9\" (UniqueName: \"kubernetes.io/projected/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-kube-api-access-bgkh9\") pod \"maas-api-revoke-sub-3e31a506-n586q\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:53.015434 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:53.015382    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:53.986633 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:53.984746    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"]
Nov 03 20:19:54.022666 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:19:54.022560    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a84c63_1cd5_49bb_bb9d_e980eefbc4b1.slice/crio-bbc32976cfa0650ca23a87d49a967e06c60319925071e25fb09ab2048ae2a0e0 WatchSource:0}: Error finding container bbc32976cfa0650ca23a87d49a967e06c60319925071e25fb09ab2048ae2a0e0: Status 404 returned error can't find the container with id bbc32976cfa0650ca23a87d49a967e06c60319925071e25fb09ab2048ae2a0e0
Nov 03 20:19:54.793801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:54.793250    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q" event={"ID":"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1","Type":"ContainerStarted","Data":"9388c71ec19e8693c2594be34fae1ad2dc8d7964e990313ee7f8e1d63cf73cc3"}
Nov 03 20:19:54.793801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:54.793351    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q" event={"ID":"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1","Type":"ContainerStarted","Data":"bbc32976cfa0650ca23a87d49a967e06c60319925071e25fb09ab2048ae2a0e0"}
Nov 03 20:19:54.809627 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:54.809575    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q" podStartSLOduration=2.809561086 podStartE2EDuration="2.809561086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:54.80775308 +0000 UTC m=+2908.077876389" watchObservedRunningTime="2026-11-03 20:19:54.809561086 +0000 UTC m=+2908.079684380"
Nov 03 20:19:55.802770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:55.802724    4531 generic.go:356] "Generic (PLEG): container finished" podID="f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1" containerID="9388c71ec19e8693c2594be34fae1ad2dc8d7964e990313ee7f8e1d63cf73cc3" exitCode=0
Nov 03 20:19:55.803323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:55.802911    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q" event={"ID":"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1","Type":"ContainerDied","Data":"9388c71ec19e8693c2594be34fae1ad2dc8d7964e990313ee7f8e1d63cf73cc3"}
Nov 03 20:19:57.448420 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.448385    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:57.567200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.564537    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-kube-api-access-bgkh9\" (UniqueName: \"kubernetes.io/projected/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-kube-api-access-bgkh9\") pod \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") "
Nov 03 20:19:57.567200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.564695    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-openshift-ca\") pod \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") "
Nov 03 20:19:57.567200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.564810    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-xks-ca\") pod \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\" (UID: \"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1\") "
Nov 03 20:19:57.580629 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.580582    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-openshift-ca" pod "f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1" (UID: "f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:57.636050 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.636011    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-kube-api-access-bgkh9" pod "f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1" (UID: "f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1"). InnerVolumeSpecName "kube-api-access-bgkh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:57.636876 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.636827    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-xks-ca" pod "f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1" (UID: "f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:57.666487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.666397    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgkh9\" (UniqueName: \"kubernetes.io/projected/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-kube-api-access-bgkh9\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:57.666487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.666432    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:57.666487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.666453    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:57.833409 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.833376    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"
Nov 03 20:19:57.833598 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.833386    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q" event={"ID":"f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1","Type":"ContainerDied","Data":"bbc32976cfa0650ca23a87d49a967e06c60319925071e25fb09ab2048ae2a0e0"}
Nov 03 20:19:57.833598 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:19:57.833509    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbc32976cfa0650ca23a87d49a967e06c60319925071e25fb09ab2048ae2a0e0"
Nov 03 20:20:04.757195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:04.756289    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg" podUID="2f1560ba-749d-4336-a0a3-8fff53d66950" containerName="main" containerID="cri-o://c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e" gracePeriod=45
Nov 03 20:20:05.312988 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.312636    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg" podUID="2f1560ba-749d-4336-a0a3-8fff53d66950" containerName="main" probeResult="failure" output="Get \"https://10.133.1.49:8000/ready\": dial tcp 10.133.1.49:8000: connect: connection refused"
Nov 03 20:20:05.798222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.794538    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:20:05.841580 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.841537    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-home\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-home\") pod \"2f1560ba-749d-4336-a0a3-8fff53d66950\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") "
Nov 03 20:20:05.871199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.848928    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-dshm\") pod \"2f1560ba-749d-4336-a0a3-8fff53d66950\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") "
Nov 03 20:20:05.871677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.871642    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1560ba-749d-4336-a0a3-8fff53d66950-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1560ba-749d-4336-a0a3-8fff53d66950-tls-certs\") pod \"2f1560ba-749d-4336-a0a3-8fff53d66950\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") "
Nov 03 20:20:05.872308 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.872280    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-model-cache\") pod \"2f1560ba-749d-4336-a0a3-8fff53d66950\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") "
Nov 03 20:20:05.872798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.872778    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-tmp-dir\") pod \"2f1560ba-749d-4336-a0a3-8fff53d66950\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") "
Nov 03 20:20:05.882204 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.872940    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1560ba-749d-4336-a0a3-8fff53d66950-kube-api-access-bzjhz\" (UniqueName: \"kubernetes.io/projected/2f1560ba-749d-4336-a0a3-8fff53d66950-kube-api-access-bzjhz\") pod \"2f1560ba-749d-4336-a0a3-8fff53d66950\" (UID: \"2f1560ba-749d-4336-a0a3-8fff53d66950\") "
Nov 03 20:20:05.893504 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.847314    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-home" pod "2f1560ba-749d-4336-a0a3-8fff53d66950" (UID: "2f1560ba-749d-4336-a0a3-8fff53d66950"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:05.909771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.872678    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-model-cache" pod "2f1560ba-749d-4336-a0a3-8fff53d66950" (UID: "2f1560ba-749d-4336-a0a3-8fff53d66950"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:05.909771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.873444    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-tmp-dir" pod "2f1560ba-749d-4336-a0a3-8fff53d66950" (UID: "2f1560ba-749d-4336-a0a3-8fff53d66950"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:05.909771 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.909404    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-dshm" pod "2f1560ba-749d-4336-a0a3-8fff53d66950" (UID: "2f1560ba-749d-4336-a0a3-8fff53d66950"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:05.922303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.922260    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1560ba-749d-4336-a0a3-8fff53d66950-tls-certs" pod "2f1560ba-749d-4336-a0a3-8fff53d66950" (UID: "2f1560ba-749d-4336-a0a3-8fff53d66950"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:05.922303 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.922268    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1560ba-749d-4336-a0a3-8fff53d66950-kube-api-access-bzjhz" pod "2f1560ba-749d-4336-a0a3-8fff53d66950" (UID: "2f1560ba-749d-4336-a0a3-8fff53d66950"). InnerVolumeSpecName "kube-api-access-bzjhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:05.980104 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.978642    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-tdppc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:05.980104 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.978744    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-tdppc" podUID="de3afce5-6597-4f19-8f0f-d4a695f4af39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:05.995401 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.995365    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:05.995401 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.995403    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f1560ba-749d-4336-a0a3-8fff53d66950-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:05.995609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.995418    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:05.995609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.995433    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:05.995609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.995449    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzjhz\" (UniqueName: \"kubernetes.io/projected/2f1560ba-749d-4336-a0a3-8fff53d66950-kube-api-access-bzjhz\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:05.995609 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:05.995463    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f1560ba-749d-4336-a0a3-8fff53d66950-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:06.102335 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.102282    4531 generic.go:356] "Generic (PLEG): container finished" podID="2f1560ba-749d-4336-a0a3-8fff53d66950" containerID="c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e" exitCode=0
Nov 03 20:20:06.102335 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.102328    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"
Nov 03 20:20:06.102607 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.102369    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg" event={"ID":"2f1560ba-749d-4336-a0a3-8fff53d66950","Type":"ContainerDied","Data":"c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e"}
Nov 03 20:20:06.102607 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.102426    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg" event={"ID":"2f1560ba-749d-4336-a0a3-8fff53d66950","Type":"ContainerDied","Data":"93d53154d40293cadbf132a928eb03d1d22824c3bf550da63d589cd2923f46e4"}
Nov 03 20:20:06.102607 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.102449    4531 scope.go:122] "RemoveContainer" containerID="c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e"
Nov 03 20:20:06.112129 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.112110    4531 scope.go:122] "RemoveContainer" containerID="c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e"
Nov 03 20:20:06.112418 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:20:06.112397    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e\": container with ID starting with c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e not found: ID does not exist" containerID="c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e"
Nov 03 20:20:06.112515 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.112425    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e"} err="failed to get container status \"c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e\": rpc error: code = NotFound desc = could not find container \"c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e\": container with ID starting with c3b825db00451a6f01be8315c423ba8b1213ba2c2a4d4e39c76be18f0bf46b4e not found: ID does not exist"
Nov 03 20:20:06.123715 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.123686    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"]
Nov 03 20:20:06.126746 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.126723    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a0ada1a8-kserve-7d8fb46b85-25qbg"]
Nov 03 20:20:06.186590 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.186506    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n" podUID="19f810dc-b163-4c59-b17a-8b11e5b8fef0" containerName="main" containerID="cri-o://62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0" gracePeriod=45
Nov 03 20:20:06.446263 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.446162    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:20:06.601796 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.601752    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tmp-dir\") pod \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") "
Nov 03 20:20:06.601997 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.601904    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-home\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-home\") pod \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") "
Nov 03 20:20:06.601997 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.601989    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f810dc-b163-4c59-b17a-8b11e5b8fef0-kube-api-access-w6dgl\" (UniqueName: \"kubernetes.io/projected/19f810dc-b163-4c59-b17a-8b11e5b8fef0-kube-api-access-w6dgl\") pod \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") "
Nov 03 20:20:06.602169 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.602052    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-dshm\") pod \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") "
Nov 03 20:20:06.602169 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.602121    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-model-cache\") pod \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") "
Nov 03 20:20:06.602169 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.602146    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tmp-dir" pod "19f810dc-b163-4c59-b17a-8b11e5b8fef0" (UID: "19f810dc-b163-4c59-b17a-8b11e5b8fef0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:06.602380 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.602166    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tls-certs\" (UniqueName: \"kubernetes.io/secret/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tls-certs\") pod \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\" (UID: \"19f810dc-b163-4c59-b17a-8b11e5b8fef0\") "
Nov 03 20:20:06.602441 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.602401    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-home" pod "19f810dc-b163-4c59-b17a-8b11e5b8fef0" (UID: "19f810dc-b163-4c59-b17a-8b11e5b8fef0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:06.602615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.602593    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tmp-dir\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:06.602699 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.602621    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-home\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:06.603006 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.602971    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-model-cache" pod "19f810dc-b163-4c59-b17a-8b11e5b8fef0" (UID: "19f810dc-b163-4c59-b17a-8b11e5b8fef0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:06.605239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.605098    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f810dc-b163-4c59-b17a-8b11e5b8fef0-kube-api-access-w6dgl" pod "19f810dc-b163-4c59-b17a-8b11e5b8fef0" (UID: "19f810dc-b163-4c59-b17a-8b11e5b8fef0"). InnerVolumeSpecName "kube-api-access-w6dgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:06.605469 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.605426    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tls-certs" pod "19f810dc-b163-4c59-b17a-8b11e5b8fef0" (UID: "19f810dc-b163-4c59-b17a-8b11e5b8fef0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:06.608450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.608430    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-dshm" pod "19f810dc-b163-4c59-b17a-8b11e5b8fef0" (UID: "19f810dc-b163-4c59-b17a-8b11e5b8fef0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:06.703588 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.703483    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6dgl\" (UniqueName: \"kubernetes.io/projected/19f810dc-b163-4c59-b17a-8b11e5b8fef0-kube-api-access-w6dgl\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:06.703588 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.703528    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-dshm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:06.703588 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.703561    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19f810dc-b163-4c59-b17a-8b11e5b8fef0-model-cache\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:06.703588 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.703575    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f810dc-b163-4c59-b17a-8b11e5b8fef0-tls-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:06.860212 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:06.860122    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1560ba-749d-4336-a0a3-8fff53d66950" path="/var/lib/kubelet/pods/2f1560ba-749d-4336-a0a3-8fff53d66950/volumes"
Nov 03 20:20:07.178956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:07.177852    4531 generic.go:356] "Generic (PLEG): container finished" podID="19f810dc-b163-4c59-b17a-8b11e5b8fef0" containerID="62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0" exitCode=0
Nov 03 20:20:07.178956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:07.177918    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n" event={"ID":"19f810dc-b163-4c59-b17a-8b11e5b8fef0","Type":"ContainerDied","Data":"62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0"}
Nov 03 20:20:07.178956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:07.177958    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n" event={"ID":"19f810dc-b163-4c59-b17a-8b11e5b8fef0","Type":"ContainerDied","Data":"371acbd45015d2eb7efc2ab3f415ff873cb79baf3ea7e85f8f9d2eb8fa7110d7"}
Nov 03 20:20:07.178956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:07.177984    4531 scope.go:122] "RemoveContainer" containerID="62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0"
Nov 03 20:20:07.178956 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:07.178102    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"
Nov 03 20:20:07.194864 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:07.194825    4531 scope.go:122] "RemoveContainer" containerID="62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0"
Nov 03 20:20:07.195460 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:20:07.195422    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0\": container with ID starting with 62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0 not found: ID does not exist" containerID="62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0"
Nov 03 20:20:07.195573 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:07.195474    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0"} err="failed to get container status \"62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0\": rpc error: code = NotFound desc = could not find container \"62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0\": container with ID starting with 62fc75a2a15491a2118bfc4cbf0883c653433ca1be3450d136c7c1edc351cbd0 not found: ID does not exist"
Nov 03 20:20:07.207229 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:07.205055    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"]
Nov 03 20:20:07.208841 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:07.208815    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a0ada1a8-kserve-7c4b4bf5d9-ds25n"]
Nov 03 20:20:08.903198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:08.887386    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f810dc-b163-4c59-b17a-8b11e5b8fef0" path="/var/lib/kubelet/pods/19f810dc-b163-4c59-b17a-8b11e5b8fef0/volumes"
Nov 03 20:20:12.969026 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.968935    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"]
Nov 03 20:20:12.969565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.969541    4531 cpu_manager.go:414] "containerMap: removing container" podUID="19f810dc-b163-4c59-b17a-8b11e5b8fef0" containerName="main"
Nov 03 20:20:12.969650 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.969569    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f810dc-b163-4c59-b17a-8b11e5b8fef0" containerName="main"
Nov 03 20:20:12.969650 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.969598    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2f1560ba-749d-4336-a0a3-8fff53d66950" containerName="main"
Nov 03 20:20:12.969650 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.969607    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1560ba-749d-4336-a0a3-8fff53d66950" containerName="main"
Nov 03 20:20:12.969819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.969652    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1" containerName="revoke-keys"
Nov 03 20:20:12.969819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.969661    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1" containerName="revoke-keys"
Nov 03 20:20:12.969819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.969760    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f810dc-b163-4c59-b17a-8b11e5b8fef0" containerName="main"
Nov 03 20:20:12.969819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.969777    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1560ba-749d-4336-a0a3-8fff53d66950" containerName="main"
Nov 03 20:20:12.969819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.969793    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1" containerName="revoke-keys"
Nov 03 20:20:12.974300 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.974276    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:12.976976 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.976949    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:20:12.977152 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.976984    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:20:12.977429 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.977410    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:20:12.977554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.977504    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:20:12.978654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:12.978630    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"]
Nov 03 20:20:13.013656 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:13.013623    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cbl4\" (UniqueName: \"kubernetes.io/projected/dffb5411-ab53-4a7b-856c-1c18959128f4-kube-api-access-8cbl4\") pod \"maas-api-revoke-sub-c4efbe8c-cjzj2\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:13.013820 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:13.013664    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4efbe8c-cjzj2\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:13.013820 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:13.013741    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4efbe8c-cjzj2\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:13.114537 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:13.114499    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cbl4\" (UniqueName: \"kubernetes.io/projected/dffb5411-ab53-4a7b-856c-1c18959128f4-kube-api-access-8cbl4\") pod \"maas-api-revoke-sub-c4efbe8c-cjzj2\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:13.114722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:13.114544    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4efbe8c-cjzj2\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:13.114722 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:13.114580    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4efbe8c-cjzj2\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:13.115349 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:13.115322    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4efbe8c-cjzj2\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:13.117053 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:13.117029    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4efbe8c-cjzj2\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:13.121849 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:13.121829    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cbl4\" (UniqueName: \"kubernetes.io/projected/dffb5411-ab53-4a7b-856c-1c18959128f4-kube-api-access-8cbl4\") pod \"maas-api-revoke-sub-c4efbe8c-cjzj2\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:13.294819 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:13.294717    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:13.566853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:13.566825    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"]
Nov 03 20:20:13.568992 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:20:13.568961    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffb5411_ab53_4a7b_856c_1c18959128f4.slice/crio-5c036a553ded6eaee1be33154b18a3d32af3f3c7050c098f0508207cccfe50f4 WatchSource:0}: Error finding container 5c036a553ded6eaee1be33154b18a3d32af3f3c7050c098f0508207cccfe50f4: Status 404 returned error can't find the container with id 5c036a553ded6eaee1be33154b18a3d32af3f3c7050c098f0508207cccfe50f4
Nov 03 20:20:14.230754 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:14.230707    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2" event={"ID":"dffb5411-ab53-4a7b-856c-1c18959128f4","Type":"ContainerStarted","Data":"86d98fd0d94fbd206ca9eefd19224b9330680746dea2045dccabc2c8fc4df468"}
Nov 03 20:20:14.231200 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:14.230772    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2" event={"ID":"dffb5411-ab53-4a7b-856c-1c18959128f4","Type":"ContainerStarted","Data":"5c036a553ded6eaee1be33154b18a3d32af3f3c7050c098f0508207cccfe50f4"}
Nov 03 20:20:14.247531 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:14.247480    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2" podStartSLOduration=2.247465574 podStartE2EDuration="2.247465574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:14.245334244 +0000 UTC m=+2927.515457538" watchObservedRunningTime="2026-11-03 20:20:14.247465574 +0000 UTC m=+2927.517588866"
Nov 03 20:20:15.237395 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:15.237361    4531 generic.go:356] "Generic (PLEG): container finished" podID="dffb5411-ab53-4a7b-856c-1c18959128f4" containerID="86d98fd0d94fbd206ca9eefd19224b9330680746dea2045dccabc2c8fc4df468" exitCode=0
Nov 03 20:20:15.237915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:15.237444    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2" event={"ID":"dffb5411-ab53-4a7b-856c-1c18959128f4","Type":"ContainerDied","Data":"86d98fd0d94fbd206ca9eefd19224b9330680746dea2045dccabc2c8fc4df468"}
Nov 03 20:20:16.378668 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:16.378639    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:16.449202 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:16.449142    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-openshift-ca\") pod \"dffb5411-ab53-4a7b-856c-1c18959128f4\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") "
Nov 03 20:20:16.449420 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:16.449248    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-xks-ca\") pod \"dffb5411-ab53-4a7b-856c-1c18959128f4\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") "
Nov 03 20:20:16.449420 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:16.449339    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffb5411-ab53-4a7b-856c-1c18959128f4-kube-api-access-8cbl4\" (UniqueName: \"kubernetes.io/projected/dffb5411-ab53-4a7b-856c-1c18959128f4-kube-api-access-8cbl4\") pod \"dffb5411-ab53-4a7b-856c-1c18959128f4\" (UID: \"dffb5411-ab53-4a7b-856c-1c18959128f4\") "
Nov 03 20:20:16.449624 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:16.449598    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-openshift-ca" pod "dffb5411-ab53-4a7b-856c-1c18959128f4" (UID: "dffb5411-ab53-4a7b-856c-1c18959128f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:16.451412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:16.451384    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-xks-ca" pod "dffb5411-ab53-4a7b-856c-1c18959128f4" (UID: "dffb5411-ab53-4a7b-856c-1c18959128f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:16.451526 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:16.451439    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffb5411-ab53-4a7b-856c-1c18959128f4-kube-api-access-8cbl4" pod "dffb5411-ab53-4a7b-856c-1c18959128f4" (UID: "dffb5411-ab53-4a7b-856c-1c18959128f4"). InnerVolumeSpecName "kube-api-access-8cbl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:16.550685 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:16.550584    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-xks-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:16.550685 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:16.550631    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cbl4\" (UniqueName: \"kubernetes.io/projected/dffb5411-ab53-4a7b-856c-1c18959128f4-kube-api-access-8cbl4\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:16.550685 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:16.550646    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffb5411-ab53-4a7b-856c-1c18959128f4-maas-api-openshift-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:17.247862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:17.247820    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2" event={"ID":"dffb5411-ab53-4a7b-856c-1c18959128f4","Type":"ContainerDied","Data":"5c036a553ded6eaee1be33154b18a3d32af3f3c7050c098f0508207cccfe50f4"}
Nov 03 20:20:17.247862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:17.247848    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"
Nov 03 20:20:17.248155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:17.247875    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c036a553ded6eaee1be33154b18a3d32af3f3c7050c098f0508207cccfe50f4"
Nov 03 20:20:17.717344 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:17.717313    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:20.572310 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:20:20.572271    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:20:25.121318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.121277    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"]
Nov 03 20:20:25.121890 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.121868    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dffb5411-ab53-4a7b-856c-1c18959128f4" containerName="revoke-keys"
Nov 03 20:20:25.121982 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.121894    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffb5411-ab53-4a7b-856c-1c18959128f4" containerName="revoke-keys"
Nov 03 20:20:25.122080 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.122066    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffb5411-ab53-4a7b-856c-1c18959128f4" containerName="revoke-keys"
Nov 03 20:20:25.125545 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.125521    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.131029 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.130992    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:20:25.133017 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.132982    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"]
Nov 03 20:20:25.135341 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.135058    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 20:20:25.135485 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.135370    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:20:25.135643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.135626    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rj6j4\""
Nov 03 20:20:25.136724 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.136704    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:20:25.138125 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.138102    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"]
Nov 03 20:20:25.141901 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.141877    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.151015 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.150706    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"]
Nov 03 20:20:25.242274 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.242163    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443c93ae-c90c-405a-9d78-4e5c536d7fa3-openshift-service-ca\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.242449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.242298    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f497a91-d139-4b18-bf46-6f62d0155ed8-openshift-service-ca\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.242449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.242321    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443c93ae-c90c-405a-9d78-4e5c536d7fa3-maas-api-tls\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.242449 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.242418    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-metrics-certs\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.242568 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.242448    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77p4\" (UniqueName: \"kubernetes.io/projected/9f497a91-d139-4b18-bf46-6f62d0155ed8-kube-api-access-q77p4\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.242568 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.242475    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxftt\" (UniqueName: \"kubernetes.io/projected/443c93ae-c90c-405a-9d78-4e5c536d7fa3-kube-api-access-dxftt\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.242642 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.242577    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/443c93ae-c90c-405a-9d78-4e5c536d7fa3-metrics-certs\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.242642 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.242597    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-maas-api-tls\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.344157 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.344121    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/443c93ae-c90c-405a-9d78-4e5c536d7fa3-metrics-certs\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.344389 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.344190    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-maas-api-tls\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.344389 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.344278    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443c93ae-c90c-405a-9d78-4e5c536d7fa3-openshift-service-ca\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.344389 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.344305    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f497a91-d139-4b18-bf46-6f62d0155ed8-openshift-service-ca\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.344389 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.344322    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443c93ae-c90c-405a-9d78-4e5c536d7fa3-maas-api-tls\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.344618 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.344416    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-metrics-certs\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.344618 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.344456    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77p4\" (UniqueName: \"kubernetes.io/projected/9f497a91-d139-4b18-bf46-6f62d0155ed8-kube-api-access-q77p4\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.344618 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.344499    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxftt\" (UniqueName: \"kubernetes.io/projected/443c93ae-c90c-405a-9d78-4e5c536d7fa3-kube-api-access-dxftt\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.345117 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.345060    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443c93ae-c90c-405a-9d78-4e5c536d7fa3-openshift-service-ca\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.345117 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.345085    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f497a91-d139-4b18-bf46-6f62d0155ed8-openshift-service-ca\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.347276 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.347249    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/443c93ae-c90c-405a-9d78-4e5c536d7fa3-metrics-certs\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.347365 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.347283    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-metrics-certs\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.347365 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.347260    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-maas-api-tls\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.347365 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.347343    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443c93ae-c90c-405a-9d78-4e5c536d7fa3-maas-api-tls\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.351921 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.351896    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxftt\" (UniqueName: \"kubernetes.io/projected/443c93ae-c90c-405a-9d78-4e5c536d7fa3-kube-api-access-dxftt\") pod \"maas-api-7d9ffb8588-hqsbh\" (UID: \"443c93ae-c90c-405a-9d78-4e5c536d7fa3\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.352020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.351952    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77p4\" (UniqueName: \"kubernetes.io/projected/9f497a91-d139-4b18-bf46-6f62d0155ed8-kube-api-access-q77p4\") pod \"maas-api-6599bcbcd4-wgtvs\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.443156 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.443116    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:25.459507 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.459472    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:25.728431 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.728403    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"]
Nov 03 20:20:25.729601 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:20:25.729571    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f497a91_d139_4b18_bf46_6f62d0155ed8.slice/crio-d8609b95b19e597fb5d941a2f1049b86eaddde4da31af6fe4f49180943c9b5cb WatchSource:0}: Error finding container d8609b95b19e597fb5d941a2f1049b86eaddde4da31af6fe4f49180943c9b5cb: Status 404 returned error can't find the container with id d8609b95b19e597fb5d941a2f1049b86eaddde4da31af6fe4f49180943c9b5cb
Nov 03 20:20:25.730912 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.730894    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:20:25.750917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:25.750890    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"]
Nov 03 20:20:25.752897 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:20:25.752868    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443c93ae_c90c_405a_9d78_4e5c536d7fa3.slice/crio-110d8868a9bf674201e53586b90c270fb25f5ac72375b7a230aaa456b381d79e WatchSource:0}: Error finding container 110d8868a9bf674201e53586b90c270fb25f5ac72375b7a230aaa456b381d79e: Status 404 returned error can't find the container with id 110d8868a9bf674201e53586b90c270fb25f5ac72375b7a230aaa456b381d79e
Nov 03 20:20:26.295623 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:26.295567    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs" event={"ID":"9f497a91-d139-4b18-bf46-6f62d0155ed8","Type":"ContainerStarted","Data":"12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09"}
Nov 03 20:20:26.296060 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:26.295644    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:26.296060 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:26.295666    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs" event={"ID":"9f497a91-d139-4b18-bf46-6f62d0155ed8","Type":"ContainerStarted","Data":"d8609b95b19e597fb5d941a2f1049b86eaddde4da31af6fe4f49180943c9b5cb"}
Nov 03 20:20:26.297236 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:26.297211    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh" event={"ID":"443c93ae-c90c-405a-9d78-4e5c536d7fa3","Type":"ContainerStarted","Data":"0ba72c7a82b1339faca2a66a4536776324c9ea609bc6c0d7f24a0361066c0dfe"}
Nov 03 20:20:26.297337 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:26.297259    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:26.297337 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:26.297277    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh" event={"ID":"443c93ae-c90c-405a-9d78-4e5c536d7fa3","Type":"ContainerStarted","Data":"110d8868a9bf674201e53586b90c270fb25f5ac72375b7a230aaa456b381d79e"}
Nov 03 20:20:26.312545 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:26.312489    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs" podStartSLOduration=0.919065565 podStartE2EDuration="1.312472162s" totalImagesPullingTime="393.406597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:26.311203963 +0000 UTC m=+2939.581327260" watchObservedRunningTime="2026-11-03 20:20:26.312472162 +0000 UTC m=+2939.582595460"
Nov 03 20:20:26.326974 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:26.326920    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh" podStartSLOduration=0.919302758 podStartE2EDuration="1.326909395s" totalImagesPullingTime="407.606637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:26.325536418 +0000 UTC m=+2939.595659712" watchObservedRunningTime="2026-11-03 20:20:26.326909395 +0000 UTC m=+2939.597032690"
Nov 03 20:20:31.801038 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:31.801003    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:32.086114 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.086003    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"]
Nov 03 20:20:32.090954 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.090916    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.098305 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.098269    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-57cdc37c-openshift-default-dockercfg-rr8xl\""
Nov 03 20:20:32.104007 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.103977    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"]
Nov 03 20:20:32.189769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.189736    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:32.217803 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.217753    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-envoy\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.218147 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.218119    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56027237-58d5-4602-9201-4188728f9afa-istiod-ca-cert\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.218289 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.218155    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-data\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.218289 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.218230    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbzqm\" (UniqueName: \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-kube-api-access-mbzqm\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.218289 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.218264    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56027237-58d5-4602-9201-4188728f9afa-istio-podinfo\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.218491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.218308    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-certs\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.218491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.218351    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-socket\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.218491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.218381    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-credential-socket\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.218491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.218413    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-istio-token\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.307773 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.307744    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-hqsbh"
Nov 03 20:20:32.308126 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.308105    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:32.319365 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.319329    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-envoy\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.319530 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.319412    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56027237-58d5-4602-9201-4188728f9afa-istiod-ca-cert\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.319530 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.319443    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-data\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.319530 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.319497    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbzqm\" (UniqueName: \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-kube-api-access-mbzqm\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.319530 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.319525    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56027237-58d5-4602-9201-4188728f9afa-istio-podinfo\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.319770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.319619    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-certs\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.319770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.319676    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-socket\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.319770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.319710    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-credential-socket\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.319770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.319738    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-istio-token\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.320011 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.319884    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-data\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.320011 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.319963    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-certs\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.320211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.320136    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-socket\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.320211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.320198    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-credential-socket\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.320373 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.320265    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56027237-58d5-4602-9201-4188728f9afa-istiod-ca-cert\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.321765 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.321745    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-envoy\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.322009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.321991    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56027237-58d5-4602-9201-4188728f9afa-istio-podinfo\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.328463 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.328440    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-istio-token\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.328748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.328725    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbzqm\" (UniqueName: \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-kube-api-access-mbzqm\") pod \"e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") " pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.372087 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.371996    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"]
Nov 03 20:20:32.372272 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.372225    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs" podUID="9f497a91-d139-4b18-bf46-6f62d0155ed8" containerName="maas-api" containerID="cri-o://12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09" gracePeriod=30
Nov 03 20:20:32.377714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.377680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:32.410855 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.410808    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:32.641996 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.641971    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:32.699655 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.699624    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"]
Nov 03 20:20:32.701700 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:20:32.701670    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56027237_58d5_4602_9201_4188728f9afa.slice/crio-6543844856bde98c2575c099c33b9882d41f8e30be9db4c7f3ab08c15cb4cd71 WatchSource:0}: Error finding container 6543844856bde98c2575c099c33b9882d41f8e30be9db4c7f3ab08c15cb4cd71: Status 404 returned error can't find the container with id 6543844856bde98c2575c099c33b9882d41f8e30be9db4c7f3ab08c15cb4cd71
Nov 03 20:20:32.711617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.711464    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:32.711731 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.711666    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:32.711731 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.711710    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:32.724757 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.724736    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f497a91-d139-4b18-bf46-6f62d0155ed8-kube-api-access-q77p4\" (UniqueName: \"kubernetes.io/projected/9f497a91-d139-4b18-bf46-6f62d0155ed8-kube-api-access-q77p4\") pod \"9f497a91-d139-4b18-bf46-6f62d0155ed8\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") "
Nov 03 20:20:32.724847 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.724808    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-maas-api-tls\") pod \"9f497a91-d139-4b18-bf46-6f62d0155ed8\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") "
Nov 03 20:20:32.724847 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.724844    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f497a91-d139-4b18-bf46-6f62d0155ed8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f497a91-d139-4b18-bf46-6f62d0155ed8-openshift-service-ca\") pod \"9f497a91-d139-4b18-bf46-6f62d0155ed8\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") "
Nov 03 20:20:32.724967 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.724930    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-metrics-certs\") pod \"9f497a91-d139-4b18-bf46-6f62d0155ed8\" (UID: \"9f497a91-d139-4b18-bf46-6f62d0155ed8\") "
Nov 03 20:20:32.725306 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.725265    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f497a91-d139-4b18-bf46-6f62d0155ed8-openshift-service-ca" pod "9f497a91-d139-4b18-bf46-6f62d0155ed8" (UID: "9f497a91-d139-4b18-bf46-6f62d0155ed8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:32.726947 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.726919    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f497a91-d139-4b18-bf46-6f62d0155ed8-kube-api-access-q77p4" pod "9f497a91-d139-4b18-bf46-6f62d0155ed8" (UID: "9f497a91-d139-4b18-bf46-6f62d0155ed8"). InnerVolumeSpecName "kube-api-access-q77p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:32.727053 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.727019    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-maas-api-tls" pod "9f497a91-d139-4b18-bf46-6f62d0155ed8" (UID: "9f497a91-d139-4b18-bf46-6f62d0155ed8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:32.727149 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.727132    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-metrics-certs" pod "9f497a91-d139-4b18-bf46-6f62d0155ed8" (UID: "9f497a91-d139-4b18-bf46-6f62d0155ed8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:32.826405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.826355    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q77p4\" (UniqueName: \"kubernetes.io/projected/9f497a91-d139-4b18-bf46-6f62d0155ed8-kube-api-access-q77p4\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:32.826405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.826392    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:32.826405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.826407    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f497a91-d139-4b18-bf46-6f62d0155ed8-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:32.827064 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:32.826431    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f497a91-d139-4b18-bf46-6f62d0155ed8-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:33.336589 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.336557    4531 generic.go:356] "Generic (PLEG): container finished" podID="9f497a91-d139-4b18-bf46-6f62d0155ed8" containerID="12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09" exitCode=0
Nov 03 20:20:33.336769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.336636    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"
Nov 03 20:20:33.336769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.336643    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs" event={"ID":"9f497a91-d139-4b18-bf46-6f62d0155ed8","Type":"ContainerDied","Data":"12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09"}
Nov 03 20:20:33.336769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.336692    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs" event={"ID":"9f497a91-d139-4b18-bf46-6f62d0155ed8","Type":"ContainerDied","Data":"d8609b95b19e597fb5d941a2f1049b86eaddde4da31af6fe4f49180943c9b5cb"}
Nov 03 20:20:33.336769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.336724    4531 scope.go:122] "RemoveContainer" containerID="12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09"
Nov 03 20:20:33.338441 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.338415    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" event={"ID":"56027237-58d5-4602-9201-4188728f9afa","Type":"ContainerStarted","Data":"074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1"}
Nov 03 20:20:33.338565 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.338463    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" event={"ID":"56027237-58d5-4602-9201-4188728f9afa","Type":"ContainerStarted","Data":"6543844856bde98c2575c099c33b9882d41f8e30be9db4c7f3ab08c15cb4cd71"}
Nov 03 20:20:33.345942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.345922    4531 scope.go:122] "RemoveContainer" containerID="12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09"
Nov 03 20:20:33.346229 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:20:33.346207    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09\": container with ID starting with 12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09 not found: ID does not exist" containerID="12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09"
Nov 03 20:20:33.346296 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.346242    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09"} err="failed to get container status \"12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09\": rpc error: code = NotFound desc = could not find container \"12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09\": container with ID starting with 12e769aeb59bde9a11f8d4e3586b78dfda5c59e20bb6455ca22e351068a39e09 not found: ID does not exist"
Nov 03 20:20:33.355608 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.355563    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" podStartSLOduration=1.355552168 podStartE2EDuration="1.355552168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:33.354720431 +0000 UTC m=+2946.624843727" watchObservedRunningTime="2026-11-03 20:20:33.355552168 +0000 UTC m=+2946.625675463"
Nov 03 20:20:33.366239 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.366216    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"]
Nov 03 20:20:33.369674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.369654    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-wgtvs"]
Nov 03 20:20:33.411701 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.411670    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:33.417063 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:33.417042    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:34.344403 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:34.344364    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:34.345585 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:34.345564    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:34.856915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:34.856879    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f497a91-d139-4b18-bf46-6f62d0155ed8" path="/var/lib/kubelet/pods/9f497a91-d139-4b18-bf46-6f62d0155ed8/volumes"
Nov 03 20:20:35.041392 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:35.041355    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:35.266212 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:35.266144    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:36.134774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.134733    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"]
Nov 03 20:20:36.135307 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.135205    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9f497a91-d139-4b18-bf46-6f62d0155ed8" containerName="maas-api"
Nov 03 20:20:36.135307 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.135222    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f497a91-d139-4b18-bf46-6f62d0155ed8" containerName="maas-api"
Nov 03 20:20:36.135451 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.135310    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f497a91-d139-4b18-bf46-6f62d0155ed8" containerName="maas-api"
Nov 03 20:20:36.140208 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.140167    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.153388 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.153351    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"]
Nov 03 20:20:36.154482 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.154453    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-57cdc37c-serving-cert\""
Nov 03 20:20:36.201626 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.201590    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:36.225675 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.225632    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"]
Nov 03 20:20:36.229603 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.229574    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:20:36.234510 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.234428    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"]
Nov 03 20:20:36.246914 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.246886    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-57cdc37c-dockercfg-lpm7v\""
Nov 03 20:20:36.259703 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.259669    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rdc\" (UniqueName: \"kubernetes.io/projected/465e081e-8335-4ac7-8940-a5d6d49cb303-kube-api-access-v8rdc\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.259877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.259792    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-maas-api-tls\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.259877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.259855    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-metrics-certs\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.259998 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.259898    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465e081e-8335-4ac7-8940-a5d6d49cb303-openshift-service-ca\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.260862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.260838    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"]
Nov 03 20:20:36.265065 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.265044    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:20:36.279217 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.279163    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"]
Nov 03 20:20:36.300895 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.300864    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-57cdc37c\""
Nov 03 20:20:36.340537 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.340502    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:36.362406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.362107    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-maas-api-tls\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.362406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.362205    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-metrics-certs\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.362406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.362253    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvfwg\" (UniqueName: \"kubernetes.io/projected/c0d50a14-90a1-4ba4-977f-666401458181-kube-api-access-dvfwg\") pod \"payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7\" (UID: \"c0d50a14-90a1-4ba4-977f-666401458181\") " pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:20:36.362406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.362286    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465e081e-8335-4ac7-8940-a5d6d49cb303-openshift-service-ca\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.362406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.362335    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d50a14-90a1-4ba4-977f-666401458181-plugins-config-volume\") pod \"payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7\" (UID: \"c0d50a14-90a1-4ba4-977f-666401458181\") " pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:20:36.362406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.362363    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf\" (UID: \"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:20:36.362406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.362409    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rdc\" (UniqueName: \"kubernetes.io/projected/465e081e-8335-4ac7-8940-a5d6d49cb303-kube-api-access-v8rdc\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.364165 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.362456    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmvz8\" (UniqueName: \"kubernetes.io/projected/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-kube-api-access-rmvz8\") pod \"payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf\" (UID: \"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:20:36.364165 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:20:36.362629    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-serving-cert: secret "maas-api-e2e-ait-57cdc37c-serving-cert" not found
Nov 03 20:20:36.364165 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:20:36.362691    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-maas-api-tls podName:465e081e-8335-4ac7-8940-a5d6d49cb303 nodeName:}" failed. No retries permitted until 2026-11-03 20:20:36.862668061 +0000 UTC m=+2950.132791349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-maas-api-tls") pod "maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n" (UID: "465e081e-8335-4ac7-8940-a5d6d49cb303") : secret "maas-api-e2e-ait-57cdc37c-serving-cert" not found
Nov 03 20:20:36.365024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.364996    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465e081e-8335-4ac7-8940-a5d6d49cb303-openshift-service-ca\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.369719 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.369693    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-metrics-certs\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.374535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.374507    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rdc\" (UniqueName: \"kubernetes.io/projected/465e081e-8335-4ac7-8940-a5d6d49cb303-kube-api-access-v8rdc\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.463136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.463027    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d50a14-90a1-4ba4-977f-666401458181-plugins-config-volume\") pod \"payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7\" (UID: \"c0d50a14-90a1-4ba4-977f-666401458181\") " pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:20:36.463136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.463068    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf\" (UID: \"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:20:36.463136 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.463121    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmvz8\" (UniqueName: \"kubernetes.io/projected/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-kube-api-access-rmvz8\") pod \"payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf\" (UID: \"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:20:36.463487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.463253    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfwg\" (UniqueName: \"kubernetes.io/projected/c0d50a14-90a1-4ba4-977f-666401458181-kube-api-access-dvfwg\") pod \"payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7\" (UID: \"c0d50a14-90a1-4ba4-977f-666401458181\") " pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:20:36.464066 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.464040    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d50a14-90a1-4ba4-977f-666401458181-plugins-config-volume\") pod \"payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7\" (UID: \"c0d50a14-90a1-4ba4-977f-666401458181\") " pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:20:36.464131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.464040    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf\" (UID: \"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:20:36.470681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.470656    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfwg\" (UniqueName: \"kubernetes.io/projected/c0d50a14-90a1-4ba4-977f-666401458181-kube-api-access-dvfwg\") pod \"payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7\" (UID: \"c0d50a14-90a1-4ba4-977f-666401458181\") " pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:20:36.470792 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.470735    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmvz8\" (UniqueName: \"kubernetes.io/projected/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-kube-api-access-rmvz8\") pod \"payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf\" (UID: \"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:20:36.486619 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.486593    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:36.550421 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.550380    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:20:36.585636 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.585530    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:20:36.793406 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.793307    4531 scope.go:122] "RemoveContainer" containerID="397ff534a280be06ead3bdf74025d72d13b16785f919fb26f0d576717f6b86c6"
Nov 03 20:20:36.802677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.802654    4531 scope.go:122] "RemoveContainer" containerID="84d8b1b03d527c28d0a98d801677921ddf4e176e379099624488de3c0a74fef8"
Nov 03 20:20:36.846292 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.846266    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"]
Nov 03 20:20:36.849461 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:20:36.849428    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ac0c4b_c9c9_4bdb_9aa4_eea91554d5be.slice/crio-23bf87ca992e1bbfc6eeda620ce80fbe7af4fe94959aec98b3a35d3cb17b7077 WatchSource:0}: Error finding container 23bf87ca992e1bbfc6eeda620ce80fbe7af4fe94959aec98b3a35d3cb17b7077: Status 404 returned error can't find the container with id 23bf87ca992e1bbfc6eeda620ce80fbe7af4fe94959aec98b3a35d3cb17b7077
Nov 03 20:20:36.878345 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.878315    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-maas-api-tls\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.881336 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.881310    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-maas-api-tls\") pod \"maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:36.882910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:36.882888    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"]
Nov 03 20:20:36.886058 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:20:36.886026    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d50a14_90a1_4ba4_977f_666401458181.slice/crio-1027d14f9e29aa4a8480dcdf81616eba073b851f36791c904423a8fc88f4aaf1 WatchSource:0}: Error finding container 1027d14f9e29aa4a8480dcdf81616eba073b851f36791c904423a8fc88f4aaf1: Status 404 returned error can't find the container with id 1027d14f9e29aa4a8480dcdf81616eba073b851f36791c904423a8fc88f4aaf1
Nov 03 20:20:37.061224 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.061117    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:37.145533 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.128423    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:37.340351 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.340320    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"]
Nov 03 20:20:37.342784 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:20:37.342756    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465e081e_8335_4ac7_8940_a5d6d49cb303.slice/crio-d44ec169b30e9f64f30c2f346ab9f1227dedd0dd5d0e9c23d6cdccce245380ec WatchSource:0}: Error finding container d44ec169b30e9f64f30c2f346ab9f1227dedd0dd5d0e9c23d6cdccce245380ec: Status 404 returned error can't find the container with id d44ec169b30e9f64f30c2f346ab9f1227dedd0dd5d0e9c23d6cdccce245380ec
Nov 03 20:20:37.368128 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.368093    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf" event={"ID":"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be","Type":"ContainerStarted","Data":"00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72"}
Nov 03 20:20:37.368336 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.368162    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf" event={"ID":"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be","Type":"ContainerStarted","Data":"23bf87ca992e1bbfc6eeda620ce80fbe7af4fe94959aec98b3a35d3cb17b7077"}
Nov 03 20:20:37.368336 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.368270    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:20:37.369476 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.369455    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n" event={"ID":"465e081e-8335-4ac7-8940-a5d6d49cb303","Type":"ContainerStarted","Data":"d44ec169b30e9f64f30c2f346ab9f1227dedd0dd5d0e9c23d6cdccce245380ec"}
Nov 03 20:20:37.371107 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.371083    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7" event={"ID":"c0d50a14-90a1-4ba4-977f-666401458181","Type":"ContainerStarted","Data":"fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b"}
Nov 03 20:20:37.371244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.371120    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7" event={"ID":"c0d50a14-90a1-4ba4-977f-666401458181","Type":"ContainerStarted","Data":"1027d14f9e29aa4a8480dcdf81616eba073b851f36791c904423a8fc88f4aaf1"}
Nov 03 20:20:37.371244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.371225    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:20:37.383994 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.383938    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf" podStartSLOduration=1.383922989 podStartE2EDuration="1.383922989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:37.382474115 +0000 UTC m=+2950.652597411" watchObservedRunningTime="2026-11-03 20:20:37.383922989 +0000 UTC m=+2950.654046285"
Nov 03 20:20:37.396697 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.396624    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7" podStartSLOduration=1.396609873 podStartE2EDuration="1.396609873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:37.39656657 +0000 UTC m=+2950.666689856" watchObservedRunningTime="2026-11-03 20:20:37.396609873 +0000 UTC m=+2950.666733169"
Nov 03 20:20:37.831884 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:37.831834    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:38.019638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:38.019553    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"]
Nov 03 20:20:38.023638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:38.023610    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dxn4q"]
Nov 03 20:20:38.380131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:38.380088    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n" event={"ID":"465e081e-8335-4ac7-8940-a5d6d49cb303","Type":"ContainerStarted","Data":"e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1"}
Nov 03 20:20:38.380646 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:38.380467    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:38.396417 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:38.396363    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n" podStartSLOduration=2.013979756 podStartE2EDuration="2.396349651s" totalImagesPullingTime="382.369895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:38.39466656 +0000 UTC m=+2951.664789855" watchObservedRunningTime="2026-11-03 20:20:38.396349651 +0000 UTC m=+2951.666472958"
Nov 03 20:20:38.527105 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:38.527071    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:38.858917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:38.858822    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f984ea-d900-44a0-b2bd-80ff5903e18e" path="/var/lib/kubelet/pods/f0f984ea-d900-44a0-b2bd-80ff5903e18e/volumes"
Nov 03 20:20:39.372869 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:39.372833    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:40.152261 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:40.152219    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:41.030391 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:41.030352    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:41.835582 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:41.835537    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:42.622606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:42.622568    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:43.457199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:43.448778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:44.266795 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:44.266759    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:44.392774 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:44.392735    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:20:45.065924 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:45.065877    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:45.948258 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:45.948202    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:46.622214 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:46.618651    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:47.392451 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:47.392409    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:48.017487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:48.017457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:48.383486 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:48.383455    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:20:48.383661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:48.383650    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:20:48.435915 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:48.435883    4531 generic.go:356] "Generic (PLEG): container finished" podID="56027237-58d5-4602-9201-4188728f9afa" containerID="074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1" exitCode=0
Nov 03 20:20:48.436127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:48.435959    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" event={"ID":"56027237-58d5-4602-9201-4188728f9afa","Type":"ContainerDied","Data":"074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1"}
Nov 03 20:20:48.436474 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:48.436453    4531 scope.go:122] "RemoveContainer" containerID="074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1"
Nov 03 20:20:48.437751 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:48.437726    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:48.437812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:48.437800    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:48.437854 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:48.437834    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:48.789816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:48.789727    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:49.348197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:49.344299    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:49.467080 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:49.467044    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" event={"ID":"56027237-58d5-4602-9201-4188728f9afa","Type":"ContainerStarted","Data":"ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728"}
Nov 03 20:20:49.532391 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:49.532358    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:50.321235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:50.321201    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:50.411129 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:50.411089    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:50.412467 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:50.412435    4531 patch_prober.go:29] interesting pod/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 03 20:20:50.412591 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:50.412502    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 03 20:20:50.589324 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:20:50.589233    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:20:51.330121 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:51.330084    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:51.411770 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:51.411730    4531 patch_prober.go:29] interesting pod/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 03 20:20:51.412219 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:51.411798    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 03 20:20:52.232007 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:52.231972    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:52.411431 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:52.411394    4531 patch_prober.go:29] interesting pod/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 03 20:20:52.411648 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:52.411459    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 03 20:20:53.074694 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:53.074644    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:53.411404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:53.411361    4531 patch_prober.go:29] interesting pod/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body=
Nov 03 20:20:53.411608 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:53.411429    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused"
Nov 03 20:20:53.921268 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:53.921224    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:54.466024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:54.465984    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:54.466748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:54.466723    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:54.468122 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:54.468096    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:20:54.711818 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:54.711772    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:55.525155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:55.525105    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:56.230799 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:56.230734    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:56.920257 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:56.920166    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:57.660858 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:57.660823    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:58.325773 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:58.325733    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:59.024492 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:59.024457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:59.725681 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:20:59.725647    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:00.416069 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:00.416034    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:00.744256 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:00.744139    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"]
Nov 03 20:21:00.757434 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:00.757403    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"]
Nov 03 20:21:00.860754 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:00.860720    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:00.896889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:00.896846    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"]
Nov 03 20:21:00.897113 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:00.897086    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n" podUID="465e081e-8335-4ac7-8940-a5d6d49cb303" containerName="maas-api" containerID="cri-o://e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1" gracePeriod=30
Nov 03 20:21:00.994899 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:00.994802    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:01.174416 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.174387    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:21:01.200193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.200143    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:01.266283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.266154    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465e081e-8335-4ac7-8940-a5d6d49cb303-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465e081e-8335-4ac7-8940-a5d6d49cb303-openshift-service-ca\") pod \"465e081e-8335-4ac7-8940-a5d6d49cb303\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") "
Nov 03 20:21:01.266283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.266240    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465e081e-8335-4ac7-8940-a5d6d49cb303-kube-api-access-v8rdc\" (UniqueName: \"kubernetes.io/projected/465e081e-8335-4ac7-8940-a5d6d49cb303-kube-api-access-v8rdc\") pod \"465e081e-8335-4ac7-8940-a5d6d49cb303\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") "
Nov 03 20:21:01.266283 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.266264    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-maas-api-tls\") pod \"465e081e-8335-4ac7-8940-a5d6d49cb303\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") "
Nov 03 20:21:01.266711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.266679    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465e081e-8335-4ac7-8940-a5d6d49cb303-openshift-service-ca" pod "465e081e-8335-4ac7-8940-a5d6d49cb303" (UID: "465e081e-8335-4ac7-8940-a5d6d49cb303"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:01.266861 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.266795    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-metrics-certs\" (UniqueName: \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-metrics-certs\") pod \"465e081e-8335-4ac7-8940-a5d6d49cb303\" (UID: \"465e081e-8335-4ac7-8940-a5d6d49cb303\") "
Nov 03 20:21:01.267476 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.267450    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/465e081e-8335-4ac7-8940-a5d6d49cb303-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:01.269337 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.268716    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-metrics-certs" pod "465e081e-8335-4ac7-8940-a5d6d49cb303" (UID: "465e081e-8335-4ac7-8940-a5d6d49cb303"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:01.269445 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.269117    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465e081e-8335-4ac7-8940-a5d6d49cb303-kube-api-access-v8rdc" pod "465e081e-8335-4ac7-8940-a5d6d49cb303" (UID: "465e081e-8335-4ac7-8940-a5d6d49cb303"). InnerVolumeSpecName "kube-api-access-v8rdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:01.269445 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.269163    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-maas-api-tls" pod "465e081e-8335-4ac7-8940-a5d6d49cb303" (UID: "465e081e-8335-4ac7-8940-a5d6d49cb303"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:01.368415 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.368373    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8rdc\" (UniqueName: \"kubernetes.io/projected/465e081e-8335-4ac7-8940-a5d6d49cb303-kube-api-access-v8rdc\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:01.368415 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.368409    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:01.368415 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.368419    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/465e081e-8335-4ac7-8940-a5d6d49cb303-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:01.529855 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.529756    4531 generic.go:356] "Generic (PLEG): container finished" podID="465e081e-8335-4ac7-8940-a5d6d49cb303" containerID="e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1" exitCode=0
Nov 03 20:21:01.529855 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.529825    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"
Nov 03 20:21:01.530080 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.529850    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n" event={"ID":"465e081e-8335-4ac7-8940-a5d6d49cb303","Type":"ContainerDied","Data":"e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1"}
Nov 03 20:21:01.530080 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.529909    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n" event={"ID":"465e081e-8335-4ac7-8940-a5d6d49cb303","Type":"ContainerDied","Data":"d44ec169b30e9f64f30c2f346ab9f1227dedd0dd5d0e9c23d6cdccce245380ec"}
Nov 03 20:21:01.530080 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.529937    4531 scope.go:122] "RemoveContainer" containerID="e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1"
Nov 03 20:21:01.540310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.540291    4531 scope.go:122] "RemoveContainer" containerID="e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1"
Nov 03 20:21:01.540594 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:21:01.540574    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1\": container with ID starting with e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1 not found: ID does not exist" containerID="e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1"
Nov 03 20:21:01.540641 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.540602    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1"} err="failed to get container status \"e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1\": rpc error: code = NotFound desc = could not find container \"e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1\": container with ID starting with e2a8d82958b3ce1c0a4528ecfab051571a50ad6d450940d138910e11a1bc01f1 not found: ID does not exist"
Nov 03 20:21:01.551156 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.551127    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"]
Nov 03 20:21:01.553223 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.553201    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-57cdc37c-6bc7b6fb7d-8pj2n"]
Nov 03 20:21:01.785905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.785818    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:01.908973 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:01.908934    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:02.857335 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:02.857300    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465e081e-8335-4ac7-8940-a5d6d49cb303" path="/var/lib/kubelet/pods/465e081e-8335-4ac7-8940-a5d6d49cb303/volumes"
Nov 03 20:21:05.744894 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:05.744837    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7" podUID="c0d50a14-90a1-4ba4-977f-666401458181" containerName="payload-processing" containerID="cri-o://fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b" gracePeriod=25
Nov 03 20:21:05.758450 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:05.758414    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf" podUID="47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be" containerName="payload-pre-processing" containerID="cri-o://00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72" gracePeriod=25
Nov 03 20:21:06.012867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.012839    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:21:06.051853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.051824    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:21:06.118792 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.118756    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d50a14-90a1-4ba4-977f-666401458181-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d50a14-90a1-4ba4-977f-666401458181-plugins-config-volume\") pod \"c0d50a14-90a1-4ba4-977f-666401458181\" (UID: \"c0d50a14-90a1-4ba4-977f-666401458181\") "
Nov 03 20:21:06.118971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.118803    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-kube-api-access-rmvz8\" (UniqueName: \"kubernetes.io/projected/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-kube-api-access-rmvz8\") pod \"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be\" (UID: \"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be\") "
Nov 03 20:21:06.118971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.118914    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d50a14-90a1-4ba4-977f-666401458181-kube-api-access-dvfwg\" (UniqueName: \"kubernetes.io/projected/c0d50a14-90a1-4ba4-977f-666401458181-kube-api-access-dvfwg\") pod \"c0d50a14-90a1-4ba4-977f-666401458181\" (UID: \"c0d50a14-90a1-4ba4-977f-666401458181\") "
Nov 03 20:21:06.119051 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.118971    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-plugins-config-volume\") pod \"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be\" (UID: \"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be\") "
Nov 03 20:21:06.119355 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.119250    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d50a14-90a1-4ba4-977f-666401458181-plugins-config-volume" pod "c0d50a14-90a1-4ba4-977f-666401458181" (UID: "c0d50a14-90a1-4ba4-977f-666401458181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:06.119485 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.119462    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-plugins-config-volume" pod "47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be" (UID: "47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:06.120969 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.120949    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-kube-api-access-rmvz8" pod "47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be" (UID: "47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be"). InnerVolumeSpecName "kube-api-access-rmvz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:06.121027 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.121007    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d50a14-90a1-4ba4-977f-666401458181-kube-api-access-dvfwg" pod "c0d50a14-90a1-4ba4-977f-666401458181" (UID: "c0d50a14-90a1-4ba4-977f-666401458181"). InnerVolumeSpecName "kube-api-access-dvfwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:06.219863 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.219822    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvfwg\" (UniqueName: \"kubernetes.io/projected/c0d50a14-90a1-4ba4-977f-666401458181-kube-api-access-dvfwg\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:06.219863 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.219857    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:06.219863 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.219867    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d50a14-90a1-4ba4-977f-666401458181-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:06.220123 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.219878    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmvz8\" (UniqueName: \"kubernetes.io/projected/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be-kube-api-access-rmvz8\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:06.556331 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.556286    4531 generic.go:356] "Generic (PLEG): container finished" podID="c0d50a14-90a1-4ba4-977f-666401458181" containerID="fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b" exitCode=0
Nov 03 20:21:06.556509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.556342    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"
Nov 03 20:21:06.556509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.556377    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7" event={"ID":"c0d50a14-90a1-4ba4-977f-666401458181","Type":"ContainerDied","Data":"fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b"}
Nov 03 20:21:06.556509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.556437    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7" event={"ID":"c0d50a14-90a1-4ba4-977f-666401458181","Type":"ContainerDied","Data":"1027d14f9e29aa4a8480dcdf81616eba073b851f36791c904423a8fc88f4aaf1"}
Nov 03 20:21:06.556509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.556460    4531 scope.go:122] "RemoveContainer" containerID="fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b"
Nov 03 20:21:06.557857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.557829    4531 generic.go:356] "Generic (PLEG): container finished" podID="47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be" containerID="00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72" exitCode=0
Nov 03 20:21:06.557987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.557858    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf" event={"ID":"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be","Type":"ContainerDied","Data":"00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72"}
Nov 03 20:21:06.557987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.557888    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"
Nov 03 20:21:06.557987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.557905    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf" event={"ID":"47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be","Type":"ContainerDied","Data":"23bf87ca992e1bbfc6eeda620ce80fbe7af4fe94959aec98b3a35d3cb17b7077"}
Nov 03 20:21:06.567378 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.567360    4531 scope.go:122] "RemoveContainer" containerID="fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b"
Nov 03 20:21:06.567791 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:21:06.567771    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b\": container with ID starting with fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b not found: ID does not exist" containerID="fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b"
Nov 03 20:21:06.567876 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.567804    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b"} err="failed to get container status \"fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b\": rpc error: code = NotFound desc = could not find container \"fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b\": container with ID starting with fcf9af367e82a445593752e8e66db2a111661cd05e912f8f424c10690143f24b not found: ID does not exist"
Nov 03 20:21:06.567876 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.567831    4531 scope.go:122] "RemoveContainer" containerID="00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72"
Nov 03 20:21:06.576931 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.576910    4531 scope.go:122] "RemoveContainer" containerID="00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72"
Nov 03 20:21:06.577280 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:21:06.577257    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72\": container with ID starting with 00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72 not found: ID does not exist" containerID="00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72"
Nov 03 20:21:06.577578 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.577286    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72"} err="failed to get container status \"00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72\": rpc error: code = NotFound desc = could not find container \"00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72\": container with ID starting with 00306201a204a60a2c263eb38ab735abeba2e1d39e4240ba22e0be0d5807de72 not found: ID does not exist"
Nov 03 20:21:06.577578 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.577557    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"]
Nov 03 20:21:06.581705 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.581680    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57cdc37c-5fdfd5b8cc-f8ch7"]
Nov 03 20:21:06.591825 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.591803    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"]
Nov 03 20:21:06.597674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.597652    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-57cdc37c-6774ff54cf-vdnvf"]
Nov 03 20:21:06.858625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.858545    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be" path="/var/lib/kubelet/pods/47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be/volumes"
Nov 03 20:21:06.859010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.858938    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d50a14-90a1-4ba4-977f-666401458181" path="/var/lib/kubelet/pods/c0d50a14-90a1-4ba4-977f-666401458181/volumes"
Nov 03 20:21:06.920598 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.920561    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"]
Nov 03 20:21:06.920798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.920776    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy" containerID="cri-o://ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728" gracePeriod=30
Nov 03 20:21:06.995948 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:06.995911    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:08.202498 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.202460    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:08.544750 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.544655    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"]
Nov 03 20:21:08.545120 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.545107    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c0d50a14-90a1-4ba4-977f-666401458181" containerName="payload-processing"
Nov 03 20:21:08.545211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.545121    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d50a14-90a1-4ba4-977f-666401458181" containerName="payload-processing"
Nov 03 20:21:08.545211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.545135    4531 cpu_manager.go:414] "containerMap: removing container" podUID="465e081e-8335-4ac7-8940-a5d6d49cb303" containerName="maas-api"
Nov 03 20:21:08.545211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.545140    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465e081e-8335-4ac7-8940-a5d6d49cb303" containerName="maas-api"
Nov 03 20:21:08.545211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.545163    4531 cpu_manager.go:414] "containerMap: removing container" podUID="47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be" containerName="payload-pre-processing"
Nov 03 20:21:08.545211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.545181    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be" containerName="payload-pre-processing"
Nov 03 20:21:08.545491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.545239    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="465e081e-8335-4ac7-8940-a5d6d49cb303" containerName="maas-api"
Nov 03 20:21:08.545491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.545250    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ac0c4b-c9c9-4bdb-9aa4-eea91554d5be" containerName="payload-pre-processing"
Nov 03 20:21:08.545491 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.545262    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d50a14-90a1-4ba4-977f-666401458181" containerName="payload-processing"
Nov 03 20:21:08.549928 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.549903    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.558889 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.558853    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"]
Nov 03 20:21:08.561804 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.561774    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-17f6e3-openshift-default-dockercfg-bmfdv\""
Nov 03 20:21:08.621044 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.621009    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:08.645877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.645839    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2dfk\" (UniqueName: \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-kube-api-access-v2dfk\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.646083 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.645885    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-socket\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.646083 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.646012    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-podinfo\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.646205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.646109    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-envoy\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.646205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.646135    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-certs\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.646205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.646163    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-data\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.646356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.646259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-token\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.646356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.646290    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8a5b91-17c6-4642-9d89-b366de74aea1-istiod-ca-cert\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.646356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.646323    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-credential-socket\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.747522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.747475    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-socket\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.747744 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.747539    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-podinfo\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.747744 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.747572    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-envoy\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.747744 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.747590    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-certs\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.747744 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.747725    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-data\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.748001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.747808    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-token\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.748001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.747863    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8a5b91-17c6-4642-9d89-b366de74aea1-istiod-ca-cert\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.748001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.747920    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-credential-socket\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.748001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.747971    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2dfk\" (UniqueName: \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-kube-api-access-v2dfk\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.748287 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.748019    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-certs\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.748287 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.748117    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-data\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.748411 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.748357    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-credential-socket\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.748587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.748558    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-socket\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.748704 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.748659    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8a5b91-17c6-4642-9d89-b366de74aea1-istiod-ca-cert\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.749987 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.749966    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-envoy\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.750248 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.750228    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-podinfo\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.755509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.755489    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-token\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.755707 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.755688    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2dfk\" (UniqueName: \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-kube-api-access-v2dfk\") pod \"e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") " pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:08.757198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.757166    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:08.868935 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:08.868853    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:09.155548 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:09.155509    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"]
Nov 03 20:21:09.159773 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:21:09.159738    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8a5b91_17c6_4642_9d89_b366de74aea1.slice/crio-5ebe061634f4c448df86222025d9cab52943a7060c096dfef5e234905754cb11 WatchSource:0}: Error finding container 5ebe061634f4c448df86222025d9cab52943a7060c096dfef5e234905754cb11: Status 404 returned error can't find the container with id 5ebe061634f4c448df86222025d9cab52943a7060c096dfef5e234905754cb11
Nov 03 20:21:09.162425 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:09.162383    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:09.162509 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:09.162477    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:09.162547 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:09.162519    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:09.468880 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:09.468788    4531 patch_prober.go:29] interesting pod/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 03 20:21:09.468880 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:09.468858    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:21:09.580160 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:09.580125    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" event={"ID":"3e8a5b91-17c6-4642-9d89-b366de74aea1","Type":"ContainerStarted","Data":"e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b"}
Nov 03 20:21:09.580364 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:09.580213    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" event={"ID":"3e8a5b91-17c6-4642-9d89-b366de74aea1","Type":"ContainerStarted","Data":"5ebe061634f4c448df86222025d9cab52943a7060c096dfef5e234905754cb11"}
Nov 03 20:21:09.598927 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:09.598873    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" podStartSLOduration=1.598857178 podStartE2EDuration="1.598857178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:09.597113713 +0000 UTC m=+2982.867237009" watchObservedRunningTime="2026-11-03 20:21:09.598857178 +0000 UTC m=+2982.868980473"
Nov 03 20:21:09.869920 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:09.869833    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:10.874731 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:10.874696    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:11.590504 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:11.590474    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:11.591691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:11.591666    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:11.797700 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:11.797669    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:12.027625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.027592    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:12.173549 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.173523    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:21:12.286962 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.286862    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56027237-58d5-4602-9201-4188728f9afa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56027237-58d5-4602-9201-4188728f9afa-istiod-ca-cert\") pod \"56027237-58d5-4602-9201-4188728f9afa\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") "
Nov 03 20:21:12.286962 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.286955    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-credential-socket\") pod \"56027237-58d5-4602-9201-4188728f9afa\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") "
Nov 03 20:21:12.287277 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.286982    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56027237-58d5-4602-9201-4188728f9afa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56027237-58d5-4602-9201-4188728f9afa-istio-podinfo\") pod \"56027237-58d5-4602-9201-4188728f9afa\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") "
Nov 03 20:21:12.287277 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287005    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-certs\") pod \"56027237-58d5-4602-9201-4188728f9afa\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") "
Nov 03 20:21:12.287277 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287071    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-istio-token\" (UniqueName: \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-istio-token\") pod \"56027237-58d5-4602-9201-4188728f9afa\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") "
Nov 03 20:21:12.287277 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287118    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-socket\") pod \"56027237-58d5-4602-9201-4188728f9afa\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") "
Nov 03 20:21:12.287277 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287210    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-kube-api-access-mbzqm\" (UniqueName: \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-kube-api-access-mbzqm\") pod \"56027237-58d5-4602-9201-4188728f9afa\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") "
Nov 03 20:21:12.287277 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287266    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-data\") pod \"56027237-58d5-4602-9201-4188728f9afa\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") "
Nov 03 20:21:12.287658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287300    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-envoy\") pod \"56027237-58d5-4602-9201-4188728f9afa\" (UID: \"56027237-58d5-4602-9201-4188728f9afa\") "
Nov 03 20:21:12.287658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287345    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-credential-socket" pod "56027237-58d5-4602-9201-4188728f9afa" (UID: "56027237-58d5-4602-9201-4188728f9afa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:12.287658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287449    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-socket" pod "56027237-58d5-4602-9201-4188728f9afa" (UID: "56027237-58d5-4602-9201-4188728f9afa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:12.287658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287460    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-certs" pod "56027237-58d5-4602-9201-4188728f9afa" (UID: "56027237-58d5-4602-9201-4188728f9afa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:12.287658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287492    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56027237-58d5-4602-9201-4188728f9afa-istiod-ca-cert" pod "56027237-58d5-4602-9201-4188728f9afa" (UID: "56027237-58d5-4602-9201-4188728f9afa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:12.287658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287612    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-data" pod "56027237-58d5-4602-9201-4188728f9afa" (UID: "56027237-58d5-4602-9201-4188728f9afa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:12.287993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287796    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:12.287993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287820    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:12.287993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287835    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56027237-58d5-4602-9201-4188728f9afa-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:12.287993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287850    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:12.287993 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.287863    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:12.289519 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.289493    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-envoy" pod "56027237-58d5-4602-9201-4188728f9afa" (UID: "56027237-58d5-4602-9201-4188728f9afa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:12.289631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.289525    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-kube-api-access-mbzqm" pod "56027237-58d5-4602-9201-4188728f9afa" (UID: "56027237-58d5-4602-9201-4188728f9afa"). InnerVolumeSpecName "kube-api-access-mbzqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:12.289757 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.289735    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-istio-token" pod "56027237-58d5-4602-9201-4188728f9afa" (UID: "56027237-58d5-4602-9201-4188728f9afa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:12.289853 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.289776    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56027237-58d5-4602-9201-4188728f9afa-istio-podinfo" pod "56027237-58d5-4602-9201-4188728f9afa" (UID: "56027237-58d5-4602-9201-4188728f9afa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:21:12.388778 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.388735    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbzqm\" (UniqueName: \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-kube-api-access-mbzqm\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:12.388778 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.388774    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56027237-58d5-4602-9201-4188728f9afa-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:12.389037 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.388789    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56027237-58d5-4602-9201-4188728f9afa-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:12.389037 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.388802    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56027237-58d5-4602-9201-4188728f9afa-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:12.597707 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.597610    4531 generic.go:356] "Generic (PLEG): container finished" podID="56027237-58d5-4602-9201-4188728f9afa" containerID="ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728" exitCode=0
Nov 03 20:21:12.597707 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.597669    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"
Nov 03 20:21:12.597707 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.597695    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" event={"ID":"56027237-58d5-4602-9201-4188728f9afa","Type":"ContainerDied","Data":"ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728"}
Nov 03 20:21:12.598002 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.597756    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw" event={"ID":"56027237-58d5-4602-9201-4188728f9afa","Type":"ContainerDied","Data":"6543844856bde98c2575c099c33b9882d41f8e30be9db4c7f3ab08c15cb4cd71"}
Nov 03 20:21:12.598002 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.597786    4531 scope.go:122] "RemoveContainer" containerID="ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728"
Nov 03 20:21:12.609482 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.609459    4531 scope.go:122] "RemoveContainer" containerID="074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1"
Nov 03 20:21:12.618498 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.618472    4531 scope.go:122] "RemoveContainer" containerID="ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728"
Nov 03 20:21:12.618813 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:21:12.618790    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728\": container with ID starting with ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728 not found: ID does not exist" containerID="ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728"
Nov 03 20:21:12.618890 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.618822    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728"} err="failed to get container status \"ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728\": rpc error: code = NotFound desc = could not find container \"ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728\": container with ID starting with ad813c1e45fc5439e5d4d8082da858ed76253cf5cf73fb13d255b10c21316728 not found: ID does not exist"
Nov 03 20:21:12.618890 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.618842    4531 scope.go:122] "RemoveContainer" containerID="074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1"
Nov 03 20:21:12.619099 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:21:12.619084    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1\": container with ID starting with 074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1 not found: ID does not exist" containerID="074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1"
Nov 03 20:21:12.619142 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.619103    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1"} err="failed to get container status \"074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1\": rpc error: code = NotFound desc = could not find container \"074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1\": container with ID starting with 074ff0edf1992d7965814c34738891f6f262704a5c1adb648405f54ea6750fc1 not found: ID does not exist"
Nov 03 20:21:12.620037 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.620017    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"]
Nov 03 20:21:12.623654 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.623629    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-57cdc37c-openshift-default-7cbc575ddc-pn9xw"]
Nov 03 20:21:12.862597 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.862514    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56027237-58d5-4602-9201-4188728f9afa" path="/var/lib/kubelet/pods/56027237-58d5-4602-9201-4188728f9afa/volumes"
Nov 03 20:21:12.916900 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.916859    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"]
Nov 03 20:21:12.917314 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.917298    4531 cpu_manager.go:414] "containerMap: removing container" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy"
Nov 03 20:21:12.917314 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.917314    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy"
Nov 03 20:21:12.917432 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.917341    4531 cpu_manager.go:414] "containerMap: removing container" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy"
Nov 03 20:21:12.917432 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.917346    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy"
Nov 03 20:21:12.917432 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.917408    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy"
Nov 03 20:21:12.917432 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.917420    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="56027237-58d5-4602-9201-4188728f9afa" containerName="istio-proxy"
Nov 03 20:21:12.921992 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.921970    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:12.933164 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.933071    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-17f6e3-serving-cert\""
Nov 03 20:21:12.939212 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.935081    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"]
Nov 03 20:21:12.995116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.995074    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzkjw\" (UniqueName: \"kubernetes.io/projected/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-kube-api-access-hzkjw\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:12.995444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.995162    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:12.995444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.995259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:12.995444 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:12.995290    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:13.042471 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.042438    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:13.073490 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.073455    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"]
Nov 03 20:21:13.077163 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.077145    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:21:13.081640 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.081609    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-17f6e3\""
Nov 03 20:21:13.087072 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.086343    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-17f6e3-dockercfg-rjlpb\""
Nov 03 20:21:13.090377 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.090351    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"]
Nov 03 20:21:13.096767 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.096727    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzkjw\" (UniqueName: \"kubernetes.io/projected/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-kube-api-access-hzkjw\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:13.097230 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.097199    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:13.097391 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.097360    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:13.097468 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.097401    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:13.098461 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.098440    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:13.100038 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.100009    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:13.100607 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.100586    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:13.106515 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.106495    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzkjw\" (UniqueName: \"kubernetes.io/projected/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-kube-api-access-hzkjw\") pod \"maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:13.112428 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.112401    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"]
Nov 03 20:21:13.116952 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.116895    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:21:13.124773 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.124746    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"]
Nov 03 20:21:13.195548 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.195505    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:13.198525 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.198487    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0620874b-f1cc-416b-8f5d-957d66ad68e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv\" (UID: \"0620874b-f1cc-416b-8f5d-957d66ad68e6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:21:13.198886 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.198858    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv79r\" (UniqueName: \"kubernetes.io/projected/0620874b-f1cc-416b-8f5d-957d66ad68e6-kube-api-access-wv79r\") pod \"payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv\" (UID: \"0620874b-f1cc-416b-8f5d-957d66ad68e6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:21:13.199009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.198891    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f84c753-a49d-4140-a3f4-ced528c54ded-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh\" (UID: \"3f84c753-a49d-4140-a3f4-ced528c54ded\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:21:13.199009 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.198943    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdfg2\" (UniqueName: \"kubernetes.io/projected/3f84c753-a49d-4140-a3f4-ced528c54ded-kube-api-access-cdfg2\") pod \"payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh\" (UID: \"3f84c753-a49d-4140-a3f4-ced528c54ded\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:21:13.240358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.240324    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:13.305877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.299919    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdfg2\" (UniqueName: \"kubernetes.io/projected/3f84c753-a49d-4140-a3f4-ced528c54ded-kube-api-access-cdfg2\") pod \"payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh\" (UID: \"3f84c753-a49d-4140-a3f4-ced528c54ded\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:21:13.305877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.300063    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0620874b-f1cc-416b-8f5d-957d66ad68e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv\" (UID: \"0620874b-f1cc-416b-8f5d-957d66ad68e6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:21:13.305877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.300157    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv79r\" (UniqueName: \"kubernetes.io/projected/0620874b-f1cc-416b-8f5d-957d66ad68e6-kube-api-access-wv79r\") pod \"payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv\" (UID: \"0620874b-f1cc-416b-8f5d-957d66ad68e6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:21:13.305877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.300221    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f84c753-a49d-4140-a3f4-ced528c54ded-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh\" (UID: \"3f84c753-a49d-4140-a3f4-ced528c54ded\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:21:13.305877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.301272    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f84c753-a49d-4140-a3f4-ced528c54ded-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh\" (UID: \"3f84c753-a49d-4140-a3f4-ced528c54ded\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:21:13.305877 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.301273    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0620874b-f1cc-416b-8f5d-957d66ad68e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv\" (UID: \"0620874b-f1cc-416b-8f5d-957d66ad68e6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:21:13.311370 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.311324    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdfg2\" (UniqueName: \"kubernetes.io/projected/3f84c753-a49d-4140-a3f4-ced528c54ded-kube-api-access-cdfg2\") pod \"payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh\" (UID: \"3f84c753-a49d-4140-a3f4-ced528c54ded\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:21:13.311694 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.311675    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv79r\" (UniqueName: \"kubernetes.io/projected/0620874b-f1cc-416b-8f5d-957d66ad68e6-kube-api-access-wv79r\") pod \"payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv\" (UID: \"0620874b-f1cc-416b-8f5d-957d66ad68e6\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:21:13.390645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.390611    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:21:13.435553 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.435510    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:21:13.516555 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.516455    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"]
Nov 03 20:21:13.519425 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:21:13.519397    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e1e0e0_4647_48cd_b3c8_97e1ecf280e3.slice/crio-6da2dfbcfdab9421cad58f0c2a36a4618e682c00ac6d7361068b353dc87bbe95 WatchSource:0}: Error finding container 6da2dfbcfdab9421cad58f0c2a36a4618e682c00ac6d7361068b353dc87bbe95: Status 404 returned error can't find the container with id 6da2dfbcfdab9421cad58f0c2a36a4618e682c00ac6d7361068b353dc87bbe95
Nov 03 20:21:13.604919 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.604873    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw" event={"ID":"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3","Type":"ContainerStarted","Data":"6da2dfbcfdab9421cad58f0c2a36a4618e682c00ac6d7361068b353dc87bbe95"}
Nov 03 20:21:13.666010 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.665976    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"]
Nov 03 20:21:13.667995 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:21:13.667965    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0620874b_f1cc_416b_8f5d_957d66ad68e6.slice/crio-8338e7458eda3c17ddfc25daaeb0032cf947e2e4e2cd0ab6b2505a5a306f1d25 WatchSource:0}: Error finding container 8338e7458eda3c17ddfc25daaeb0032cf947e2e4e2cd0ab6b2505a5a306f1d25: Status 404 returned error can't find the container with id 8338e7458eda3c17ddfc25daaeb0032cf947e2e4e2cd0ab6b2505a5a306f1d25
Nov 03 20:21:13.718059 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.718029    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"]
Nov 03 20:21:13.722462 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:21:13.722432    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f84c753_a49d_4140_a3f4_ced528c54ded.slice/crio-6254796f24f49911819be935018434d1149a9666add792ef844e3ab58ed913ff WatchSource:0}: Error finding container 6254796f24f49911819be935018434d1149a9666add792ef844e3ab58ed913ff: Status 404 returned error can't find the container with id 6254796f24f49911819be935018434d1149a9666add792ef844e3ab58ed913ff
Nov 03 20:21:13.944100 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:13.944020    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:14.613131 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.613096    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv" event={"ID":"0620874b-f1cc-416b-8f5d-957d66ad68e6","Type":"ContainerStarted","Data":"120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f"}
Nov 03 20:21:14.613663 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.613151    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv" event={"ID":"0620874b-f1cc-416b-8f5d-957d66ad68e6","Type":"ContainerStarted","Data":"8338e7458eda3c17ddfc25daaeb0032cf947e2e4e2cd0ab6b2505a5a306f1d25"}
Nov 03 20:21:14.613663 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.613211    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:21:14.615145 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.615117    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw" event={"ID":"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3","Type":"ContainerStarted","Data":"99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421"}
Nov 03 20:21:14.615338 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.615218    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:14.616728 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.616706    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh" event={"ID":"3f84c753-a49d-4140-a3f4-ced528c54ded","Type":"ContainerStarted","Data":"a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5"}
Nov 03 20:21:14.616842 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.616740    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh" event={"ID":"3f84c753-a49d-4140-a3f4-ced528c54ded","Type":"ContainerStarted","Data":"6254796f24f49911819be935018434d1149a9666add792ef844e3ab58ed913ff"}
Nov 03 20:21:14.616842 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.616831    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:21:14.626244 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.626201    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv" podStartSLOduration=1.626188681 podStartE2EDuration="1.626188681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:14.625728218 +0000 UTC m=+2987.895851512" watchObservedRunningTime="2026-11-03 20:21:14.626188681 +0000 UTC m=+2987.896311969"
Nov 03 20:21:14.641808 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.641755    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh" podStartSLOduration=1.641742692 podStartE2EDuration="1.641742692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:14.63958247 +0000 UTC m=+2987.909705768" watchObservedRunningTime="2026-11-03 20:21:14.641742692 +0000 UTC m=+2987.911866017"
Nov 03 20:21:14.653902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.653853    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw" podStartSLOduration=2.219978151 podStartE2EDuration="2.653841111s" totalImagesPullingTime="433.86296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:14.65330167 +0000 UTC m=+2987.923424966" watchObservedRunningTime="2026-11-03 20:21:14.653841111 +0000 UTC m=+2987.923964407"
Nov 03 20:21:14.717430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:14.717401    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:15.704358 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:15.704322    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:16.453543 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:16.453493    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:17.170699 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:17.170651    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:17.751990 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:17.751938    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:17.932606 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:17.932572    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:18.659196 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:18.651813    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:19.523395 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:19.523347    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:20.366197 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:20.358226    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:20.586265 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:21:20.586225    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:21:20.627800 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:20.627722    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:21:21.140201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:21.136257    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:22.006453 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:22.006422    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:22.735042 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:22.735008    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:23.504228 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:23.504156    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:24.244983 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:24.244939    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:24.998666 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:24.998628    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:25.615682 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:25.615567    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:25.626258 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:25.626226    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:21:25.626451 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:25.626308    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:21:25.699963 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:25.699923    4531 generic.go:356] "Generic (PLEG): container finished" podID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerID="e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b" exitCode=0
Nov 03 20:21:25.700155 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:25.699987    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" event={"ID":"3e8a5b91-17c6-4642-9d89-b366de74aea1","Type":"ContainerDied","Data":"e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b"}
Nov 03 20:21:25.700440 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:25.700424    4531 scope.go:122] "RemoveContainer" containerID="e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b"
Nov 03 20:21:25.701691 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:25.701662    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:25.701769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:25.701729    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:25.701769 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:25.701764    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:26.502847 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:26.502806    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:26.590755 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:26.590714    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:26.707326 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:26.707288    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" event={"ID":"3e8a5b91-17c6-4642-9d89-b366de74aea1","Type":"ContainerStarted","Data":"361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e"}
Nov 03 20:21:26.762404 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:26.762311    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:26.870061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:26.870018    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:26.871366 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:26.871339    4531 patch_prober.go:29] interesting pod/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 03 20:21:26.871500 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:26.871390    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 03 20:21:27.372833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:27.372798    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:27.870331 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:27.870291    4531 patch_prober.go:29] interesting pod/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 03 20:21:27.870744 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:27.870365    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 03 20:21:28.150923 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:28.150889    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:28.869318 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:28.869273    4531 patch_prober.go:29] interesting pod/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 03 20:21:28.869544 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:28.869350    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 03 20:21:29.059970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:29.059917    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:29.859198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:29.845832    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:29.869998 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:29.869958    4531 patch_prober.go:29] interesting pod/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 03 20:21:29.870201 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:29.870028    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 03 20:21:30.767265 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:30.767220    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:30.869798 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:30.869765    4531 patch_prober.go:29] interesting pod/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body=
Nov 03 20:21:30.869995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:30.869819    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused"
Nov 03 20:21:31.622971 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:31.622931    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:31.906837 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:31.906793    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:31.907756 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:31.907701    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:31.908756 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:31.908735    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:21:32.408564 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:32.408518    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:32.632405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:32.632374    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:21:32.660576 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:32.660501    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:21:33.112364 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:33.112270    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:33.807554 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:33.807516    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:34.516442 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:34.516410    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:35.335307 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:35.335270    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:36.126735 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:36.126682    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:36.827466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:36.827391    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:36.845586 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:36.845553    4531 scope.go:122] "RemoveContainer" containerID="ca03cdf37bc63b0716bcfa6a84402f1f4e269eef28462e08f86055a525331039"
Nov 03 20:21:37.619536 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:37.619499    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:38.312156 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:38.312110    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:39.024581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:39.024499    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:39.842255 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:39.842150    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:40.579488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:40.579430    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:41.304995 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:41.304953    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:42.010381 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:42.010338    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:42.804587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:42.804551    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:43.510909 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:43.510876    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:44.207325 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:44.207291    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:44.920412 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:44.920368    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:45.633897 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:45.633856    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:46.306323 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:46.306288    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:47.028838 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:47.028797    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:47.671631 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:47.671592    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:47.802833 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:47.802800    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:48.524375 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:48.524336    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:49.226235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:49.226196    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:49.910241 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:49.910202    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:50.573287 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:21:50.573245    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:21:50.748642 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:50.748602    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:51.402503 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:51.402457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:52.113979 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:52.113778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:52.812193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:52.812141    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:53.536817 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:53.536783    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:54.367917 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:54.367882    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:55.138602 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:55.138564    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:55.831337 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:55.831291    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:55.966445 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:55.966409    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:56.535990 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:56.535947    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:57.222367 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:57.222330    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:57.927742 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:57.927708    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:58.636946 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:58.636907    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:59.320867 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:21:59.320831    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:00.058185 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:00.058144    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:00.733421 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:00.733381    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:01.446978 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:01.446938    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:02.148324 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:02.148268    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:02.935857 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:02.935820    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:03.624204 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:03.624157    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:04.330704 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:04.330666    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:05.003488 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:05.003454    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:05.751942 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:05.751903    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:06.436466 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:06.436384    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:07.215109 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:07.215075    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:07.917331 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:07.917294    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:08.617040 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:08.616995    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:09.342417 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:09.342377    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:10.022144 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:10.022038    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:10.740638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:10.740606    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:11.428300 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:11.428267    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:12.125487 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:12.125454    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:12.924621 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:12.924587    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:13.633442 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:13.633400    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:14.315541 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:14.315499    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:15.022966 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:15.022926    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"]
Nov 03 20:22:15.028094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:15.028065    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2cddl"]
Nov 03 20:22:15.130632 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:15.130575    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:15.831801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:15.831720    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:16.628861 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:16.628822    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:16.857671 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:16.857640    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59dab081-a07a-4014-9a4c-8a204510cb1a" path="/var/lib/kubelet/pods/59dab081-a07a-4014-9a4c-8a204510cb1a/volumes"
Nov 03 20:22:17.336561 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:17.336460    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:17.787521 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:17.787473    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:18.134455 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:18.134410    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:18.844415 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:18.844384    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:19.618829 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:19.618791    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:20.574030 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:22:20.573985    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:22:21.989713 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:21.989683    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:22.324471 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:22.324348    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:23.021871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:23.021816    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:23.733638 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:23.733592    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:24.442340 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:24.442296    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:25.223901 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:25.223836    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:25.989457 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:25.989414    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:26.813266 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:26.813217    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:27.535641 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:27.535585    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:28.227452 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:28.227370    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:29.102990 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:29.102955    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:29.729970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:29.729932    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:30.432901 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:30.432864    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:31.124741 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:31.124705    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:31.819738 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:31.819697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:32.525887 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:32.525849    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:33.311150 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:33.311113    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:34.020408 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:34.020373    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:34.025963 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:34.025917    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"]
Nov 03 20:22:34.028090 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:34.028066    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghtf5"]
Nov 03 20:22:34.811281 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:34.811233    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:34.857288 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:34.857251    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5" path="/var/lib/kubelet/pods/b447e4d8-0aed-4ae9-938c-ca9d8c6a8ac5/volumes"
Nov 03 20:22:35.537546 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:35.537509    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:36.245598 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:36.245509    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:36.988525 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:36.988484    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:37.739246 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:37.739202    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:38.533816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:38.533784    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:39.239376 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:39.239334    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:39.925420 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:39.925388    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:40.651020 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:40.650982    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:41.363643 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:41.363600    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:42.016892 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:42.016861    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:42.723132 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:42.723084    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:43.415955 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:43.415909    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:44.130094 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:44.130043    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:44.821893 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:44.821850    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:45.531866 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:45.531804    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:46.219970 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:46.219934    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:46.919617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:46.919583    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:47.621062 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:47.620998    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:48.335532 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:48.335494    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:49.037840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:49.037799    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:50.018674 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:50.018611    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:50.573767 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:22:50.573729    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:22:50.724345 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:50.724302    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:51.414959 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:51.414921    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:52.122365 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:52.122330    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:52.925718 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:52.925662    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:53.679138 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:53.679098    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:54.450484 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:54.450452    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:55.154384 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:55.154350    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:55.913604 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:55.913534    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:56.623294 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:56.623252    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:57.330657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:57.330619    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:58.119472 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:58.119427    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:58.828465 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:58.828424    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:59.513087 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:59.513052    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:59.601394 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:59.601358    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"]
Nov 03 20:22:59.613612 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:59.613570    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"]
Nov 03 20:22:59.729425 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:59.729392    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:59.862475 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:22:59.862375    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:00.334839 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:00.334793    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:04.602240 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.602159    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh" podUID="3f84c753-a49d-4140-a3f4-ced528c54ded" containerName="payload-processing" containerID="cri-o://a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5" gracePeriod=25
Nov 03 20:23:04.614245 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.614201    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv" podUID="0620874b-f1cc-416b-8f5d-957d66ad68e6" containerName="payload-pre-processing" containerID="cri-o://120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f" gracePeriod=25
Nov 03 20:23:04.856058 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.855981    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:23:04.903953 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.903926    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:23:04.926830 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.926798    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f84c753-a49d-4140-a3f4-ced528c54ded-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f84c753-a49d-4140-a3f4-ced528c54ded-plugins-config-volume\") pod \"3f84c753-a49d-4140-a3f4-ced528c54ded\" (UID: \"3f84c753-a49d-4140-a3f4-ced528c54ded\") "
Nov 03 20:23:04.927032 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.926838    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0620874b-f1cc-416b-8f5d-957d66ad68e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0620874b-f1cc-416b-8f5d-957d66ad68e6-plugins-config-volume\") pod \"0620874b-f1cc-416b-8f5d-957d66ad68e6\" (UID: \"0620874b-f1cc-416b-8f5d-957d66ad68e6\") "
Nov 03 20:23:04.927032 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.926902    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0620874b-f1cc-416b-8f5d-957d66ad68e6-kube-api-access-wv79r\" (UniqueName: \"kubernetes.io/projected/0620874b-f1cc-416b-8f5d-957d66ad68e6-kube-api-access-wv79r\") pod \"0620874b-f1cc-416b-8f5d-957d66ad68e6\" (UID: \"0620874b-f1cc-416b-8f5d-957d66ad68e6\") "
Nov 03 20:23:04.927195 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.927072    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f84c753-a49d-4140-a3f4-ced528c54ded-kube-api-access-cdfg2\" (UniqueName: \"kubernetes.io/projected/3f84c753-a49d-4140-a3f4-ced528c54ded-kube-api-access-cdfg2\") pod \"3f84c753-a49d-4140-a3f4-ced528c54ded\" (UID: \"3f84c753-a49d-4140-a3f4-ced528c54ded\") "
Nov 03 20:23:04.927310 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.927287    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f84c753-a49d-4140-a3f4-ced528c54ded-plugins-config-volume" pod "3f84c753-a49d-4140-a3f4-ced528c54ded" (UID: "3f84c753-a49d-4140-a3f4-ced528c54ded"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:04.927430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.927358    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0620874b-f1cc-416b-8f5d-957d66ad68e6-plugins-config-volume" pod "0620874b-f1cc-416b-8f5d-957d66ad68e6" (UID: "0620874b-f1cc-416b-8f5d-957d66ad68e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:04.927482 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.927471    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f84c753-a49d-4140-a3f4-ced528c54ded-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:04.927524 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.927485    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0620874b-f1cc-416b-8f5d-957d66ad68e6-plugins-config-volume\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:04.929372 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.929341    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f84c753-a49d-4140-a3f4-ced528c54ded-kube-api-access-cdfg2" pod "3f84c753-a49d-4140-a3f4-ced528c54ded" (UID: "3f84c753-a49d-4140-a3f4-ced528c54ded"). InnerVolumeSpecName "kube-api-access-cdfg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:04.929473 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:04.929455    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0620874b-f1cc-416b-8f5d-957d66ad68e6-kube-api-access-wv79r" pod "0620874b-f1cc-416b-8f5d-957d66ad68e6" (UID: "0620874b-f1cc-416b-8f5d-957d66ad68e6"). InnerVolumeSpecName "kube-api-access-wv79r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:05.028768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.028725    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdfg2\" (UniqueName: \"kubernetes.io/projected/3f84c753-a49d-4140-a3f4-ced528c54ded-kube-api-access-cdfg2\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:05.028768 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.028766    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv79r\" (UniqueName: \"kubernetes.io/projected/0620874b-f1cc-416b-8f5d-957d66ad68e6-kube-api-access-wv79r\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:05.223633 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.223587    4531 generic.go:356] "Generic (PLEG): container finished" podID="3f84c753-a49d-4140-a3f4-ced528c54ded" containerID="a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5" exitCode=0
Nov 03 20:23:05.223840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.223655    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"
Nov 03 20:23:05.223840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.223679    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh" event={"ID":"3f84c753-a49d-4140-a3f4-ced528c54ded","Type":"ContainerDied","Data":"a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5"}
Nov 03 20:23:05.223840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.223792    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh" event={"ID":"3f84c753-a49d-4140-a3f4-ced528c54ded","Type":"ContainerDied","Data":"6254796f24f49911819be935018434d1149a9666add792ef844e3ab58ed913ff"}
Nov 03 20:23:05.223840 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.223826    4531 scope.go:122] "RemoveContainer" containerID="a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5"
Nov 03 20:23:05.225211 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.225183    4531 generic.go:356] "Generic (PLEG): container finished" podID="0620874b-f1cc-416b-8f5d-957d66ad68e6" containerID="120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f" exitCode=0
Nov 03 20:23:05.225362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.225233    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv" event={"ID":"0620874b-f1cc-416b-8f5d-957d66ad68e6","Type":"ContainerDied","Data":"120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f"}
Nov 03 20:23:05.225362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.225273    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv" event={"ID":"0620874b-f1cc-416b-8f5d-957d66ad68e6","Type":"ContainerDied","Data":"8338e7458eda3c17ddfc25daaeb0032cf947e2e4e2cd0ab6b2505a5a306f1d25"}
Nov 03 20:23:05.225362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.225233    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"
Nov 03 20:23:05.234593 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.234567    4531 scope.go:122] "RemoveContainer" containerID="a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5"
Nov 03 20:23:05.234959 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:23:05.234898    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5\": container with ID starting with a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5 not found: ID does not exist" containerID="a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5"
Nov 03 20:23:05.234959 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.234931    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5"} err="failed to get container status \"a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5\": rpc error: code = NotFound desc = could not find container \"a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5\": container with ID starting with a62fbcc9d1bbcf19a556c0b72d88dc27ca397f2564c634f29d0109239bc15bf5 not found: ID does not exist"
Nov 03 20:23:05.234959 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.234948    4531 scope.go:122] "RemoveContainer" containerID="120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f"
Nov 03 20:23:05.245362 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.245339    4531 scope.go:122] "RemoveContainer" containerID="120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f"
Nov 03 20:23:05.245653 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:23:05.245627    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f\": container with ID starting with 120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f not found: ID does not exist" containerID="120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f"
Nov 03 20:23:05.245726 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.245663    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f"} err="failed to get container status \"120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f\": rpc error: code = NotFound desc = could not find container \"120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f\": container with ID starting with 120ca348dfe5eda2948e9423088a03b1d799186ec159bfe536f583067c59952f not found: ID does not exist"
Nov 03 20:23:05.250205 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.250165    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"]
Nov 03 20:23:05.256116 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.256094    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-17f6e3-7b454fcc5f-r4jlh"]
Nov 03 20:23:05.264610 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.264588    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"]
Nov 03 20:23:05.267878 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.267858    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-17f6e3-57fd7cc66c-vqppv"]
Nov 03 20:23:05.819235 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.819197    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"]
Nov 03 20:23:05.819828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.819473    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw" podUID="84e1e0e0-4647-48cd-b3c8-97e1ecf280e3" containerName="maas-api" containerID="cri-o://99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421" gracePeriod=30
Nov 03 20:23:05.925532 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:05.925477    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:06.093862 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.093833    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:23:06.124369 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.124335    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:06.140555 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.140514    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-kube-api-access-hzkjw\" (UniqueName: \"kubernetes.io/projected/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-kube-api-access-hzkjw\") pod \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") "
Nov 03 20:23:06.140765 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.140568    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-openshift-service-ca\") pod \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") "
Nov 03 20:23:06.140765 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.140654    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-maas-api-tls\") pod \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") "
Nov 03 20:23:06.140887 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.140816    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-metrics-certs\") pod \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\" (UID: \"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3\") "
Nov 03 20:23:06.141087 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.141036    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-openshift-service-ca" pod "84e1e0e0-4647-48cd-b3c8-97e1ecf280e3" (UID: "84e1e0e0-4647-48cd-b3c8-97e1ecf280e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:06.141279 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.141199    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-openshift-service-ca\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:06.143083 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.143057    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-maas-api-tls" pod "84e1e0e0-4647-48cd-b3c8-97e1ecf280e3" (UID: "84e1e0e0-4647-48cd-b3c8-97e1ecf280e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:06.143561 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.143525    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-kube-api-access-hzkjw" pod "84e1e0e0-4647-48cd-b3c8-97e1ecf280e3" (UID: "84e1e0e0-4647-48cd-b3c8-97e1ecf280e3"). InnerVolumeSpecName "kube-api-access-hzkjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:06.143679 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.143591    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-metrics-certs" pod "84e1e0e0-4647-48cd-b3c8-97e1ecf280e3" (UID: "84e1e0e0-4647-48cd-b3c8-97e1ecf280e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:06.234905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.234806    4531 generic.go:356] "Generic (PLEG): container finished" podID="84e1e0e0-4647-48cd-b3c8-97e1ecf280e3" containerID="99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421" exitCode=0
Nov 03 20:23:06.234905 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.234881    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"
Nov 03 20:23:06.235158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.234908    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw" event={"ID":"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3","Type":"ContainerDied","Data":"99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421"}
Nov 03 20:23:06.235158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.235007    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw" event={"ID":"84e1e0e0-4647-48cd-b3c8-97e1ecf280e3","Type":"ContainerDied","Data":"6da2dfbcfdab9421cad58f0c2a36a4618e682c00ac6d7361068b353dc87bbe95"}
Nov 03 20:23:06.235158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.235131    4531 scope.go:122] "RemoveContainer" containerID="99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421"
Nov 03 20:23:06.242363 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.242287    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-maas-api-tls\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:06.242363 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.242317    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-metrics-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:06.242363 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.242336    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzkjw\" (UniqueName: \"kubernetes.io/projected/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3-kube-api-access-hzkjw\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:06.250185 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.250149    4531 scope.go:122] "RemoveContainer" containerID="99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421"
Nov 03 20:23:06.250533 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:23:06.250505    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421\": container with ID starting with 99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421 not found: ID does not exist" containerID="99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421"
Nov 03 20:23:06.250598 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.250548    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421"} err="failed to get container status \"99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421\": rpc error: code = NotFound desc = could not find container \"99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421\": container with ID starting with 99ab5c25784f8aafe9e4704e7cf96ba3ed537b717b17e590d8d63133b5b59421 not found: ID does not exist"
Nov 03 20:23:06.261824 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.261795    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"]
Nov 03 20:23:06.265645 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.265621    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-17f6e3-5c8cb58fb-9cqnw"]
Nov 03 20:23:06.317868 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.317836    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:06.680859 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.680827    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:06.857704 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.857670    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0620874b-f1cc-416b-8f5d-957d66ad68e6" path="/var/lib/kubelet/pods/0620874b-f1cc-416b-8f5d-957d66ad68e6/volumes"
Nov 03 20:23:06.858110 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.858066    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f84c753-a49d-4140-a3f4-ced528c54ded" path="/var/lib/kubelet/pods/3f84c753-a49d-4140-a3f4-ced528c54ded/volumes"
Nov 03 20:23:06.858459 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:06.858446    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e1e0e0-4647-48cd-b3c8-97e1ecf280e3" path="/var/lib/kubelet/pods/84e1e0e0-4647-48cd-b3c8-97e1ecf280e3/volumes"
Nov 03 20:23:12.480683 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:12.480602    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:12.626832 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:12.626795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:12.766626 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:12.766533    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"]
Nov 03 20:23:12.766801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:12.766766    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy" containerID="cri-o://361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e" gracePeriod=30
Nov 03 20:23:12.999400 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:12.999358    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:16.917397 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:16.917351    4531 patch_prober.go:29] interesting pod/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:23:16.918043 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:16.918008    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:23:17.823193 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:17.823121    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:18.019581 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.019557    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:23:18.118398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.118293    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-podinfo\") pod \"3e8a5b91-17c6-4642-9d89-b366de74aea1\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") "
Nov 03 20:23:18.118398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.118368    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-envoy\") pod \"3e8a5b91-17c6-4642-9d89-b366de74aea1\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") "
Nov 03 20:23:18.118398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.118395    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-credential-socket\") pod \"3e8a5b91-17c6-4642-9d89-b366de74aea1\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") "
Nov 03 20:23:18.118713 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.118435    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-token\" (UniqueName: \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-token\") pod \"3e8a5b91-17c6-4642-9d89-b366de74aea1\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") "
Nov 03 20:23:18.118713 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.118473    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8a5b91-17c6-4642-9d89-b366de74aea1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8a5b91-17c6-4642-9d89-b366de74aea1-istiod-ca-cert\") pod \"3e8a5b91-17c6-4642-9d89-b366de74aea1\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") "
Nov 03 20:23:18.118713 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.118504    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-socket\") pod \"3e8a5b91-17c6-4642-9d89-b366de74aea1\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") "
Nov 03 20:23:18.118713 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.118531    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-data\") pod \"3e8a5b91-17c6-4642-9d89-b366de74aea1\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") "
Nov 03 20:23:18.118713 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.118599    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-kube-api-access-v2dfk\" (UniqueName: \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-kube-api-access-v2dfk\") pod \"3e8a5b91-17c6-4642-9d89-b366de74aea1\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") "
Nov 03 20:23:18.118990 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.118711    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-credential-socket" pod "3e8a5b91-17c6-4642-9d89-b366de74aea1" (UID: "3e8a5b91-17c6-4642-9d89-b366de74aea1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:18.118990 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.118740    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-certs\") pod \"3e8a5b91-17c6-4642-9d89-b366de74aea1\" (UID: \"3e8a5b91-17c6-4642-9d89-b366de74aea1\") "
Nov 03 20:23:18.118990 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.118847    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-data" pod "3e8a5b91-17c6-4642-9d89-b366de74aea1" (UID: "3e8a5b91-17c6-4642-9d89-b366de74aea1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:18.119159 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.119079    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-certs" pod "3e8a5b91-17c6-4642-9d89-b366de74aea1" (UID: "3e8a5b91-17c6-4642-9d89-b366de74aea1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:18.119159 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.119106    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-credential-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:18.119159 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.119128    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-data\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:18.119372 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.119250    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-socket" pod "3e8a5b91-17c6-4642-9d89-b366de74aea1" (UID: "3e8a5b91-17c6-4642-9d89-b366de74aea1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:18.119453 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.119429    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8a5b91-17c6-4642-9d89-b366de74aea1-istiod-ca-cert" pod "3e8a5b91-17c6-4642-9d89-b366de74aea1" (UID: "3e8a5b91-17c6-4642-9d89-b366de74aea1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:18.120752 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.120723    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-podinfo" pod "3e8a5b91-17c6-4642-9d89-b366de74aea1" (UID: "3e8a5b91-17c6-4642-9d89-b366de74aea1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:23:18.120872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.120727    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-kube-api-access-v2dfk" pod "3e8a5b91-17c6-4642-9d89-b366de74aea1" (UID: "3e8a5b91-17c6-4642-9d89-b366de74aea1"). InnerVolumeSpecName "kube-api-access-v2dfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:18.120872 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.120732    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-envoy" pod "3e8a5b91-17c6-4642-9d89-b366de74aea1" (UID: "3e8a5b91-17c6-4642-9d89-b366de74aea1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:18.120949 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.120903    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-token" pod "3e8a5b91-17c6-4642-9d89-b366de74aea1" (UID: "3e8a5b91-17c6-4642-9d89-b366de74aea1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:18.219843 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.219804    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-token\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:18.219843 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.219836    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8a5b91-17c6-4642-9d89-b366de74aea1-istiod-ca-cert\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:18.219843 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.219848    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-socket\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:18.220103 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.219859    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2dfk\" (UniqueName: \"kubernetes.io/projected/3e8a5b91-17c6-4642-9d89-b366de74aea1-kube-api-access-v2dfk\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:18.220103 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.219869    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-workload-certs\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:18.220103 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.219878    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-podinfo\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:18.220103 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.219886    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8a5b91-17c6-4642-9d89-b366de74aea1-istio-envoy\") on node \"ip-10-0-131-234.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:18.351940 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.351906    4531 generic.go:356] "Generic (PLEG): container finished" podID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerID="361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e" exitCode=0
Nov 03 20:23:18.352127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.351946    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" event={"ID":"3e8a5b91-17c6-4642-9d89-b366de74aea1","Type":"ContainerDied","Data":"361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e"}
Nov 03 20:23:18.352127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.351957    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"
Nov 03 20:23:18.352127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.351989    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf" event={"ID":"3e8a5b91-17c6-4642-9d89-b366de74aea1","Type":"ContainerDied","Data":"5ebe061634f4c448df86222025d9cab52943a7060c096dfef5e234905754cb11"}
Nov 03 20:23:18.352127 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.352011    4531 scope.go:122] "RemoveContainer" containerID="361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e"
Nov 03 20:23:18.361957 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.361937    4531 scope.go:122] "RemoveContainer" containerID="e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b"
Nov 03 20:23:18.369979 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.369961    4531 scope.go:122] "RemoveContainer" containerID="361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e"
Nov 03 20:23:18.370353 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:23:18.370330    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e\": container with ID starting with 361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e not found: ID does not exist" containerID="361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e"
Nov 03 20:23:18.370430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.370365    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e"} err="failed to get container status \"361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e\": rpc error: code = NotFound desc = could not find container \"361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e\": container with ID starting with 361fd8a84fed2e6e5181ffdd214d68afb0286e9665df6bd439a688ff5025718e not found: ID does not exist"
Nov 03 20:23:18.370430 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.370390    4531 scope.go:122] "RemoveContainer" containerID="e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b"
Nov 03 20:23:18.370934 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:23:18.370686    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b\": container with ID starting with e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b not found: ID does not exist" containerID="e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b"
Nov 03 20:23:18.370934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.370714    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b"} err="failed to get container status \"e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b\": rpc error: code = NotFound desc = could not find container \"e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b\": container with ID starting with e16eace393d8e33275b216157d6c7c8e7696ad2e705433798b9751b06d6c8a7b not found: ID does not exist"
Nov 03 20:23:18.372900 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.372879    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"]
Nov 03 20:23:18.378945 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.378923    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-17f6e3-openshift-default-78ddcdcf85-pbxrf"]
Nov 03 20:23:18.857304 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:18.857270    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" path="/var/lib/kubelet/pods/3e8a5b91-17c6-4642-9d89-b366de74aea1/volumes"
Nov 03 20:23:19.559158 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:19.559122    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:20.588057 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:23:20.588003    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:23:31.428349 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:31.428315    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:36.892447 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:36.892407    4531 scope.go:122] "RemoveContainer" containerID="16d4a3bd61df05607c842c51801b4708b2d08fbbb7b1fc278c6a2ee768e38080"
Nov 03 20:23:36.904124 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:23:36.904100    4531 scope.go:122] "RemoveContainer" containerID="1e35991efdf03c7be93f72454fb344d9f849eacd435e34d4706867ed6077f9e1"
Nov 03 20:23:50.572948 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:23:50.572910    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:24:03.522405 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:03.522325    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:24:03.523286 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:03.523243    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-bg7cl" podUID="e5b033e2-ebcc-4112-99d6-d9398e5ac835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:16.564092 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:16.564059    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9bf4fd58b-7rs7s_c60aac6d-9415-41b2-8883-9eb131ded1cb/manager/0.log"
Nov 03 20:24:17.383315 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:17.383281    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86c547b4b-x4k2g_9bd3bb4d-1444-4ce0-b926-111f7bc54813/manager/0.log"
Nov 03 20:24:17.888816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:17.888776    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:18.695986 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:18.695954    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7d9ffb8588-hqsbh_443c93ae-c90c-405a-9d78-4e5c536d7fa3/maas-api/0.log"
Nov 03 20:24:19.238076 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:19.238028    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-n586q_f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1/revoke-keys/0.log"
Nov 03 20:24:19.403543 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:19.403515    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-4pw88_e73b4a5b-6f04-4b6e-9f1d-149a9aef4388/revoke-keys/0.log"
Nov 03 20:24:19.728482 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:19.728450    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c4efbe8c-cjzj2_dffb5411-ab53-4a7b-856c-1c18959128f4/revoke-keys/0.log"
Nov 03 20:24:20.572989 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:24:20.572948    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:24:23.713448 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:23.713419    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f46798c78-7wgg6_a49e8bbf-667f-40eb-942a-7407013e89b5/kube-auth-proxy/0.log"
Nov 03 20:24:24.040828 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:24.040735    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-7n48w_bd352072-e76d-4d10-93a1-a71b2e0495ab/istio-proxy/0.log"
Nov 03 20:24:24.234151 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:24.234119    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-bg7cl_e5b033e2-ebcc-4112-99d6-d9398e5ac835/istio-proxy/0.log"
Nov 03 20:24:25.237461 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:25.237407    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-t5g9v_c6beddf2-5be3-42ae-a6c7-6256834bd41f/istio-proxy/0.log"
Nov 03 20:24:25.408295 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:25.408262    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-tdppc_de3afce5-6597-4f19-8f0f-d4a695f4af39/istio-proxy/0.log"
Nov 03 20:24:27.390708 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:27.390671    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-mw5r6_c731a1ef-3377-4475-82cb-213ba543c002/main/0.log"
Nov 03 20:24:27.717049 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:27.716963    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-9hwtt_6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a/main/0.log"
Nov 03 20:24:28.053665 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:28.053574    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-nsvcq_bd23398f-be9f-410a-aa58-7f2af78f8169/main/0.log"
Nov 03 20:24:28.394703 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:28.394663    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8_3b05d47f-88b5-45ce-9af3-cbde59ec2d69/main/0.log"
Nov 03 20:24:37.693285 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:37.693251    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9bf4fd58b-7rs7s_c60aac6d-9415-41b2-8883-9eb131ded1cb/manager/0.log"
Nov 03 20:24:38.453595 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:38.453561    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86c547b4b-x4k2g_9bd3bb4d-1444-4ce0-b926-111f7bc54813/manager/0.log"
Nov 03 20:24:39.521627 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:39.521530    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7d9ffb8588-hqsbh_443c93ae-c90c-405a-9d78-4e5c536d7fa3/maas-api/0.log"
Nov 03 20:24:40.046519 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:40.046487    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-n586q_f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1/revoke-keys/0.log"
Nov 03 20:24:40.213524 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:40.213490    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-4pw88_e73b4a5b-6f04-4b6e-9f1d-149a9aef4388/revoke-keys/0.log"
Nov 03 20:24:40.531653 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:40.531624    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c4efbe8c-cjzj2_dffb5411-ab53-4a7b-856c-1c18959128f4/revoke-keys/0.log"
Nov 03 20:24:43.018871 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:43.018834    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"]
Nov 03 20:24:43.021784 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:43.021744    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4pw88"]
Nov 03 20:24:43.823711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:43.823680    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f46798c78-7wgg6_a49e8bbf-667f-40eb-942a-7407013e89b5/kube-auth-proxy/0.log"
Nov 03 20:24:44.141006 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:44.140973    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-7n48w_bd352072-e76d-4d10-93a1-a71b2e0495ab/istio-proxy/0.log"
Nov 03 20:24:44.305269 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:44.305241    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-bg7cl_e5b033e2-ebcc-4112-99d6-d9398e5ac835/istio-proxy/0.log"
Nov 03 20:24:44.856419 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:44.856382    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73b4a5b-6f04-4b6e-9f1d-149a9aef4388" path="/var/lib/kubelet/pods/e73b4a5b-6f04-4b6e-9f1d-149a9aef4388/volumes"
Nov 03 20:24:45.327742 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:45.327706    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-t5g9v_c6beddf2-5be3-42ae-a6c7-6256834bd41f/istio-proxy/0.log"
Nov 03 20:24:45.492109 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:45.492068    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-tdppc_de3afce5-6597-4f19-8f0f-d4a695f4af39/istio-proxy/0.log"
Nov 03 20:24:46.979168 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:46.979129    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-mw5r6_c731a1ef-3377-4475-82cb-213ba543c002/main/0.log"
Nov 03 20:24:47.312495 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:47.312421    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-9hwtt_6d4e2fd3-3106-4646-bb92-8f8cb9cb8c2a/main/0.log"
Nov 03 20:24:47.634370 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:47.634332    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-nsvcq_bd23398f-be9f-410a-aa58-7f2af78f8169/main/0.log"
Nov 03 20:24:47.967974 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:47.967890    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9mgd8_3b05d47f-88b5-45ce-9af3-cbde59ec2d69/main/0.log"
Nov 03 20:24:50.572997 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:24:50.572957    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:24:58.018902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:58.018867    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"]
Nov 03 20:24:58.021342 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:58.021309    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n586q"]
Nov 03 20:24:58.860224 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:24:58.860163    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1" path="/var/lib/kubelet/pods/f1a84c63-1cd5-49bb-bb9d-e980eefbc4b1/volumes"
Nov 03 20:25:17.020579 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:25:17.020546    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"]
Nov 03 20:25:17.024166 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:25:17.024141    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4efbe8c-cjzj2"]
Nov 03 20:25:18.856667 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:25:18.856631    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dffb5411-ab53-4a7b-856c-1c18959128f4" path="/var/lib/kubelet/pods/dffb5411-ab53-4a7b-856c-1c18959128f4/volumes"
Nov 03 20:25:20.145962 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:25:20.145926    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:20.571504 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:25:20.571423    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:25:22.550587 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:25:22.550551    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86c547b4b-x4k2g_9bd3bb4d-1444-4ce0-b926-111f7bc54813/manager/0.log"
Nov 03 20:25:22.811657 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:25:22.811573    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86c547b4b-x4k2g_9bd3bb4d-1444-4ce0-b926-111f7bc54813/manager/0.log"
Nov 03 20:25:50.576201 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:25:50.576136    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:26:05.738111 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:05.738069    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86c547b4b-x4k2g_9bd3bb4d-1444-4ce0-b926-111f7bc54813/manager/0.log"
Nov 03 20:26:05.986118 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:05.986069    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86c547b4b-x4k2g_9bd3bb4d-1444-4ce0-b926-111f7bc54813/manager/0.log"
Nov 03 20:26:07.062012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.061973    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fg8c7/must-gather-b749q"]
Nov 03 20:26:07.062617 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062594    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy"
Nov 03 20:26:07.062714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062621    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy"
Nov 03 20:26:07.062714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062653    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0620874b-f1cc-416b-8f5d-957d66ad68e6" containerName="payload-pre-processing"
Nov 03 20:26:07.062714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062662    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0620874b-f1cc-416b-8f5d-957d66ad68e6" containerName="payload-pre-processing"
Nov 03 20:26:07.062714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062677    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3f84c753-a49d-4140-a3f4-ced528c54ded" containerName="payload-processing"
Nov 03 20:26:07.062714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062684    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f84c753-a49d-4140-a3f4-ced528c54ded" containerName="payload-processing"
Nov 03 20:26:07.062714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062694    4531 cpu_manager.go:414] "containerMap: removing container" podUID="84e1e0e0-4647-48cd-b3c8-97e1ecf280e3" containerName="maas-api"
Nov 03 20:26:07.062714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062701    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e1e0e0-4647-48cd-b3c8-97e1ecf280e3" containerName="maas-api"
Nov 03 20:26:07.063097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062726    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy"
Nov 03 20:26:07.063097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062733    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy"
Nov 03 20:26:07.063097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062843    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy"
Nov 03 20:26:07.063097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062860    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0620874b-f1cc-416b-8f5d-957d66ad68e6" containerName="payload-pre-processing"
Nov 03 20:26:07.063097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062870    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f84c753-a49d-4140-a3f4-ced528c54ded" containerName="payload-processing"
Nov 03 20:26:07.063097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.062878    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e1e0e0-4647-48cd-b3c8-97e1ecf280e3" containerName="maas-api"
Nov 03 20:26:07.063097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.063071    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8a5b91-17c6-4642-9d89-b366de74aea1" containerName="istio-proxy"
Nov 03 20:26:07.066220 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.066193    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg8c7/must-gather-b749q"
Nov 03 20:26:07.070394 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.070371    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fg8c7\"/\"openshift-service-ca.crt\""
Nov 03 20:26:07.070714 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.070696    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fg8c7\"/\"kube-root-ca.crt\""
Nov 03 20:26:07.070801 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.070748    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fg8c7\"/\"default-dockercfg-8dkgq\""
Nov 03 20:26:07.077300 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.077277    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg8c7/must-gather-b749q"]
Nov 03 20:26:07.111938 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.111904    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:07.157963 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.157918    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c96a84d4-aa48-435a-92bd-56a98e24f413-must-gather-output\") pod \"must-gather-b749q\" (UID: \"c96a84d4-aa48-435a-92bd-56a98e24f413\") " pod="openshift-must-gather-fg8c7/must-gather-b749q"
Nov 03 20:26:07.158209 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.158073    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s447\" (UniqueName: \"kubernetes.io/projected/c96a84d4-aa48-435a-92bd-56a98e24f413-kube-api-access-5s447\") pod \"must-gather-b749q\" (UID: \"c96a84d4-aa48-435a-92bd-56a98e24f413\") " pod="openshift-must-gather-fg8c7/must-gather-b749q"
Nov 03 20:26:07.254459 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.254423    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:07.259672 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.259647    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s447\" (UniqueName: \"kubernetes.io/projected/c96a84d4-aa48-435a-92bd-56a98e24f413-kube-api-access-5s447\") pod \"must-gather-b749q\" (UID: \"c96a84d4-aa48-435a-92bd-56a98e24f413\") " pod="openshift-must-gather-fg8c7/must-gather-b749q"
Nov 03 20:26:07.259815 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.259732    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c96a84d4-aa48-435a-92bd-56a98e24f413-must-gather-output\") pod \"must-gather-b749q\" (UID: \"c96a84d4-aa48-435a-92bd-56a98e24f413\") " pod="openshift-must-gather-fg8c7/must-gather-b749q"
Nov 03 20:26:07.260106 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.260084    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c96a84d4-aa48-435a-92bd-56a98e24f413-must-gather-output\") pod \"must-gather-b749q\" (UID: \"c96a84d4-aa48-435a-92bd-56a98e24f413\") " pod="openshift-must-gather-fg8c7/must-gather-b749q"
Nov 03 20:26:07.269544 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.269515    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s447\" (UniqueName: \"kubernetes.io/projected/c96a84d4-aa48-435a-92bd-56a98e24f413-kube-api-access-5s447\") pod \"must-gather-b749q\" (UID: \"c96a84d4-aa48-435a-92bd-56a98e24f413\") " pod="openshift-must-gather-fg8c7/must-gather-b749q"
Nov 03 20:26:07.382356 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.382326    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg8c7/must-gather-b749q"
Nov 03 20:26:07.663816 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.663756    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg8c7/must-gather-b749q"]
Nov 03 20:26:07.666401 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:26:07.666368    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc96a84d4_aa48_435a_92bd_56a98e24f413.slice/crio-3fc233022cddc94e04905552ccbb2b83a8c85ba3af499b957426694e7e127514 WatchSource:0}: Error finding container 3fc233022cddc94e04905552ccbb2b83a8c85ba3af499b957426694e7e127514: Status 404 returned error can't find the container with id 3fc233022cddc94e04905552ccbb2b83a8c85ba3af499b957426694e7e127514
Nov 03 20:26:07.668206 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:07.668167    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:26:08.250297 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:08.250258    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg8c7/must-gather-b749q" event={"ID":"c96a84d4-aa48-435a-92bd-56a98e24f413","Type":"ContainerStarted","Data":"3fc233022cddc94e04905552ccbb2b83a8c85ba3af499b957426694e7e127514"}
Nov 03 20:26:09.259217 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:09.259163    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg8c7/must-gather-b749q" event={"ID":"c96a84d4-aa48-435a-92bd-56a98e24f413","Type":"ContainerStarted","Data":"2842783b0e69dd63f82a30c0407399824de2a34ad123c8b7ff5c20b0b61e5ef4"}
Nov 03 20:26:09.259648 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:09.259232    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg8c7/must-gather-b749q" event={"ID":"c96a84d4-aa48-435a-92bd-56a98e24f413","Type":"ContainerStarted","Data":"7459cdf4c645834afdc1a0a6c686259d8e40f6967af2cd4a1452c9ebe8fa799a"}
Nov 03 20:26:09.276460 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:09.276388    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fg8c7/must-gather-b749q" podStartSLOduration=1.410941532 podStartE2EDuration="2.276362671s" totalImagesPullingTime="865.421139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:09.271424974 +0000 UTC m=+3282.541548269" watchObservedRunningTime="2026-11-03 20:26:09.276362671 +0000 UTC m=+3282.546485967"
Nov 03 20:26:10.468748 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:10.468717    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vj6pd_319a92b5-37b5-40b2-88cd-5550aab72650/global-pull-secret-syncer/0.log"
Nov 03 20:26:10.629385 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:10.629351    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log"
Nov 03 20:26:10.757024 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:10.756933    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-234.us-east-2.compute.internal_d9cc75c72d3d8481cfe8602c9dd164cf/haproxy/0.log"
Nov 03 20:26:17.949658 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:17.948853    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4jl9f_0a4d21a2-f237-4ffb-88c9-1855e8b4b887/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:19.677001 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:19.676966    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94b1e02b-a67f-4a75-bf07-8eb0924dce9c/alertmanager/0.log"
Nov 03 20:26:19.707159 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:19.707130    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94b1e02b-a67f-4a75-bf07-8eb0924dce9c/config-reloader/0.log"
Nov 03 20:26:19.740359 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:19.740322    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94b1e02b-a67f-4a75-bf07-8eb0924dce9c/kube-rbac-proxy-web/0.log"
Nov 03 20:26:19.775342 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:19.775310    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94b1e02b-a67f-4a75-bf07-8eb0924dce9c/kube-rbac-proxy/0.log"
Nov 03 20:26:19.806095 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:19.806069    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94b1e02b-a67f-4a75-bf07-8eb0924dce9c/kube-rbac-proxy-metric/0.log"
Nov 03 20:26:19.835677 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:19.835650    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94b1e02b-a67f-4a75-bf07-8eb0924dce9c/prom-label-proxy/0.log"
Nov 03 20:26:19.864097 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:19.864058    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_94b1e02b-a67f-4a75-bf07-8eb0924dce9c/init-config-reloader/0.log"
Nov 03 20:26:20.074459 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.074381    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84bd6f669d-jvbjq_7668ec48-a5a4-4912-b786-604de5458489/metrics-server/0.log"
Nov 03 20:26:20.108660 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.108554    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"]
Nov 03 20:26:20.115799 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.115770    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.124198 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.122667    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"]
Nov 03 20:26:20.259012 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.258972    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-proc\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.259222 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.259129    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-sys\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.259305 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.259253    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-podres\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.259305 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.259294    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-lib-modules\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.259398 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.259339    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p82qn\" (UniqueName: \"kubernetes.io/projected/7b66c6d2-583c-4529-a445-972596a91428-kube-api-access-p82qn\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.360447 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.360361    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-proc\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.360625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.360463    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-sys\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.360625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.360521    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-podres\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.360625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.360519    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-proc\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.360625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.360549    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-lib-modules\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.360625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.360591    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82qn\" (UniqueName: \"kubernetes.io/projected/7b66c6d2-583c-4529-a445-972596a91428-kube-api-access-p82qn\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.360625 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.360596    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-sys\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.360934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.360627    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-lib-modules\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.360934 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.360675    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b66c6d2-583c-4529-a445-972596a91428-podres\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.368221 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.368162    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p82qn\" (UniqueName: \"kubernetes.io/projected/7b66c6d2-583c-4529-a445-972596a91428-kube-api-access-p82qn\") pod \"perf-node-gather-daemonset-lzp5s\" (UID: \"7b66c6d2-583c-4529-a445-972596a91428\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.400513 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.400479    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvvhh_6bb340f1-f611-44c2-99e2-550c4c49e1b0/node-exporter/0.log"
Nov 03 20:26:20.436304 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.436266    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:20.441098 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.441076    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvvhh_6bb340f1-f611-44c2-99e2-550c4c49e1b0/kube-rbac-proxy/0.log"
Nov 03 20:26:20.487333 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.487299    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvvhh_6bb340f1-f611-44c2-99e2-550c4c49e1b0/init-textfile/0.log"
Nov 03 20:26:20.581318 ip-10-0-131-234 kubenswrapper[4531]: E1103 20:26:20.581270    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rjzpc" containerName="extractor"
Nov 03 20:26:20.718990 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.718783    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"]
Nov 03 20:26:20.722947 ip-10-0-131-234 kubenswrapper[4531]: W1103 20:26:20.722909    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7b66c6d2_583c_4529_a445_972596a91428.slice/crio-f996e4feb62a018192763cdce73fdbd6911af493bb8e4f65714378b7caac5f7c WatchSource:0}: Error finding container f996e4feb62a018192763cdce73fdbd6911af493bb8e4f65714378b7caac5f7c: Status 404 returned error can't find the container with id f996e4feb62a018192763cdce73fdbd6911af493bb8e4f65714378b7caac5f7c
Nov 03 20:26:20.960974 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.960939    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-d5rvl_43f44c80-7900-4ec5-ba15-50cc0e236ca7/prometheus-operator/0.log"
Nov 03 20:26:20.984493 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:20.984419    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-d5rvl_43f44c80-7900-4ec5-ba15-50cc0e236ca7/kube-rbac-proxy/0.log"
Nov 03 20:26:21.050250 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.050218    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9fbf4f6f9-8p6lk_8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135/telemeter-client/0.log"
Nov 03 20:26:21.078599 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.078568    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9fbf4f6f9-8p6lk_8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135/reload/0.log"
Nov 03 20:26:21.108660 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.108632    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-9fbf4f6f9-8p6lk_8a2b56e8-2b00-4fdf-9ab4-e44e47cdc135/kube-rbac-proxy/0.log"
Nov 03 20:26:21.150898 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.150865    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf5978794-gs47k_b2f2b56d-0c94-496e-a57f-8d83a7df23cb/thanos-query/0.log"
Nov 03 20:26:21.179364 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.179335    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf5978794-gs47k_b2f2b56d-0c94-496e-a57f-8d83a7df23cb/kube-rbac-proxy-web/0.log"
Nov 03 20:26:21.208777 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.208750    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf5978794-gs47k_b2f2b56d-0c94-496e-a57f-8d83a7df23cb/kube-rbac-proxy/0.log"
Nov 03 20:26:21.236902 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.236829    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf5978794-gs47k_b2f2b56d-0c94-496e-a57f-8d83a7df23cb/prom-label-proxy/0.log"
Nov 03 20:26:21.269812 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.269778    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf5978794-gs47k_b2f2b56d-0c94-496e-a57f-8d83a7df23cb/kube-rbac-proxy-rules/0.log"
Nov 03 20:26:21.298157 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.298126    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bf5978794-gs47k_b2f2b56d-0c94-496e-a57f-8d83a7df23cb/kube-rbac-proxy-metrics/0.log"
Nov 03 20:26:21.402132 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.402090    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s" event={"ID":"7b66c6d2-583c-4529-a445-972596a91428","Type":"ContainerStarted","Data":"bd52adc192381bfb58bf69ec5b00722d90be47601ee00b7552772d4eaa525c15"}
Nov 03 20:26:21.402463 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.402437    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s" event={"ID":"7b66c6d2-583c-4529-a445-972596a91428","Type":"ContainerStarted","Data":"f996e4feb62a018192763cdce73fdbd6911af493bb8e4f65714378b7caac5f7c"}
Nov 03 20:26:21.403513 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:21.403490    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:23.578421 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:23.578396    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b4b4fdf6-rzlv6_825cd9ed-888e-4788-923a-4aef03b5d537/console/0.log"
Nov 03 20:26:25.522478 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:25.522427    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tzpld_1d4473c0-57b5-44a8-a467-d6a29d958c68/dns/0.log"
Nov 03 20:26:25.550207 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:25.550161    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tzpld_1d4473c0-57b5-44a8-a467-d6a29d958c68/kube-rbac-proxy/0.log"
Nov 03 20:26:25.701713 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:25.701685    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q22m7_8c63d965-5a4f-481a-8b25-0ac731629615/dns-node-resolver/0.log"
Nov 03 20:26:26.399558 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:26.399510    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gcvlj_adabc7a3-99a6-43f0-b1ed-8a0395d310a4/node-ca/0.log"
Nov 03 20:26:28.437477 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:28.437443    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s"
Nov 03 20:26:28.454189 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:28.454124    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lzp5s" podStartSLOduration=8.45411004 podStartE2EDuration="8.45411004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:21.419441583 +0000 UTC m=+3294.689564879" watchObservedRunningTime="2026-11-03 20:26:28.45411004 +0000 UTC m=+3301.724233335"
Nov 03 20:26:28.885051 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:28.885024    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f46798c78-7wgg6_a49e8bbf-667f-40eb-942a-7407013e89b5/kube-auth-proxy/0.log"
Nov 03 20:26:28.954661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:28.954628    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-7n48w_bd352072-e76d-4d10-93a1-a71b2e0495ab/istio-proxy/0.log"
Nov 03 20:26:28.989583 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:28.989552    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-bg7cl_e5b033e2-ebcc-4112-99d6-d9398e5ac835/istio-proxy/0.log"
Nov 03 20:26:29.196199 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:29.196092    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-t5g9v_c6beddf2-5be3-42ae-a6c7-6256834bd41f/istio-proxy/0.log"
Nov 03 20:26:29.230021 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:29.229989    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-tdppc_de3afce5-6597-4f19-8f0f-d4a695f4af39/istio-proxy/0.log"
Nov 03 20:26:31.377153 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:31.377115    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f46798c78-7wgg6_a49e8bbf-667f-40eb-942a-7407013e89b5/kube-auth-proxy/0.log"
Nov 03 20:26:31.418811 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:31.418781    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-7n48w_bd352072-e76d-4d10-93a1-a71b2e0495ab/istio-proxy/0.log"
Nov 03 20:26:31.438868 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:31.438836    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-bg7cl_e5b033e2-ebcc-4112-99d6-d9398e5ac835/istio-proxy/0.log"
Nov 03 20:26:31.564379 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:31.564345    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-t5g9v_c6beddf2-5be3-42ae-a6c7-6256834bd41f/istio-proxy/0.log"
Nov 03 20:26:31.584713 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:31.584684    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-tdppc_de3afce5-6597-4f19-8f0f-d4a695f4af39/istio-proxy/0.log"
Nov 03 20:26:31.857711 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:31.857684    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8pptd_99d9bea7-e81a-4fd7-8757-1eadfe124622/serve-healthcheck-canary/0.log"
Nov 03 20:26:32.696842 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:32.696387    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:26:32.724493 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:32.724463    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dqz7h_4bdd5ccb-e17b-4bd6-adb3-30c82c2836c1/ovn-acl-logging/0.log"
Nov 03 20:26:32.991238 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:32.991144    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rjzpc_9abeb54a-e625-4ad1-827e-49f5640d6288/kube-rbac-proxy/0.log"
Nov 03 20:26:33.035091 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:33.035065    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rjzpc_9abeb54a-e625-4ad1-827e-49f5640d6288/exporter/0.log"
Nov 03 20:26:33.087353 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:33.087316    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rjzpc_9abeb54a-e625-4ad1-827e-49f5640d6288/extractor/0.log"
Nov 03 20:26:33.143661 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:33.143626    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-jj9xh-5rsck_7a924521-7745-40c7-b46d-1a9e5b3375ea/insights-gathering/0.log"
Nov 03 20:26:33.191615 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:33.191570    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-wfnhm-m7rk8_64f7cc83-420d-4739-ace1-db3694d9dca1/insights-gathering/0.log"
Nov 03 20:26:35.543962 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:35.543933    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f46798c78-7wgg6_a49e8bbf-667f-40eb-942a-7407013e89b5/kube-auth-proxy/0.log"
Nov 03 20:26:35.581522 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:35.581487    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-7n48w_bd352072-e76d-4d10-93a1-a71b2e0495ab/istio-proxy/0.log"
Nov 03 20:26:35.603447 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:35.603408    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-bg7cl_e5b033e2-ebcc-4112-99d6-d9398e5ac835/istio-proxy/0.log"
Nov 03 20:26:35.677747 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:35.677713    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-bd9746b54-4zndn_36ce59a5-8947-48d8-af38-db7c046afae4/manager/0.log"
Nov 03 20:26:35.700149 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:35.700121    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-bd9746b54-4zndn_36ce59a5-8947-48d8-af38-db7c046afae4/copy-manifests/0.log"
Nov 03 20:26:35.717874 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:35.717844    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-t5g9v_c6beddf2-5be3-42ae-a6c7-6256834bd41f/istio-proxy/0.log"
Nov 03 20:26:35.737061 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:35.737033    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-tdppc_de3afce5-6597-4f19-8f0f-d4a695f4af39/istio-proxy/0.log"
Nov 03 20:26:36.985271 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:36.985233    4531 scope.go:122] "RemoveContainer" containerID="200f60c505f43f2e6b3ee831ff2d477f695c736052ab6b8ae21e86520f6871f1"
Nov 03 20:26:37.004438 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:36.997536    4531 scope.go:122] "RemoveContainer" containerID="9388c71ec19e8693c2594be34fae1ad2dc8d7964e990313ee7f8e1d63cf73cc3"
Nov 03 20:26:37.020831 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:37.020797    4531 scope.go:122] "RemoveContainer" containerID="86d98fd0d94fbd206ca9eefd19224b9330680746dea2045dccabc2c8fc4df468"
Nov 03 20:26:37.131910 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:37.131883    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-75db7bc68-7gk4z_12f77852-39af-4cc0-8d45-b264ca4ca198/manager/0.log"
Nov 03 20:26:38.030535 ip-10-0-131-234 kubenswrapper[4531]: I1103 20:26:38.030473    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9bf4fd58b-7rs7s_c60aac6d-9415-41b2-8883-9eb131ded1cb/manager/0.log"